Nov 03 22:04:23.719081 ip-10-0-141-84 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 22:04:23.719094 ip-10-0-141-84 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 22:04:23.719102 ip-10-0-141-84 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 22:04:23.719408 ip-10-0-141-84 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 22:04:33.796860 ip-10-0-141-84 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 22:04:33.796879 ip-10-0-141-84 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 723a35da0579423d8c7f6c86113ef352 -- Nov 03 22:06:29.651433 ip-10-0-141-84 systemd[1]: Starting Kubernetes Kubelet... Nov 03 22:06:30.009585 ip-10-0-141-84 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:06:30.009585 ip-10-0-141-84 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 22:06:30.009585 ip-10-0-141-84 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:06:30.009585 ip-10-0-141-84 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:06:30.013789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013689 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:30.013789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013787 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:30.013789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013791 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:30.013789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013794 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013798 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013801 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013805 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013809 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013812 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013816 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013819 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013822 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013825 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013828 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013831 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013833 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013836 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013839 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013845 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013848 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013851 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013854 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:30.013919 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013856 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013858 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013861 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013864 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013866 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013869 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013871 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013874 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013877 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013879 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013884 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013888 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013891 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013894 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013897 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013901 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013903 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013906 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013909 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013912 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:30.014384 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013915 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013918 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013920 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013923 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013927 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013931 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013943 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013946 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013949 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013952 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013955 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013958 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013960 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013963 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013966 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013969 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013972 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013975 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013978 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:30.014853 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013981 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013984 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013987 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013989 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013992 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013995 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.013998 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014001 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014004 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014007 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014010 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014013 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014016 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014019 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014021 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014024 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014027 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014030 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014032 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014035 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:30.015330 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014046 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014049 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014052 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014054 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014057 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014075 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014079 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014082 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014085 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014087 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014090 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014093 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014096 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014098 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014101 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014104 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014106 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014109 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014112 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:30.015797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014115 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014118 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014120 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014124 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014127 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.014129 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015036 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015043 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015047 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015049 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015052 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015055 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015058 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015082 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015086 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015089 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015091 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015094 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015097 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:30.016298 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015100 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015103 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015106 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015108 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015111 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015115 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015119 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015122 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015124 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015127 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015130 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015133 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015136 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015139 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015141 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015144 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015147 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015150 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015153 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015156 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:30.016776 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015159 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015162 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015165 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015167 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015170 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015173 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015176 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015179 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015183 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015186 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015188 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015191 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015194 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015197 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015199 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015203 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015206 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015209 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015211 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:30.017297 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015214 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015217 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015219 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015224 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015228 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015231 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015234 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015237 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015240 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015244 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015246 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015249 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015252 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015254 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015257 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015260 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015262 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015265 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015268 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:30.017770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015271 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015273 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015276 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015279 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015282 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015284 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015288 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015291 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015293 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015296 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015299 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015302 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015305 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015307 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015310 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015313 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015315 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015319 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015322 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015325 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:30.018856 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015327 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015331 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015337 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015340 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015342 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015345 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015348 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015351 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015353 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015356 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015359 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015361 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015364 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015366 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015369 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015942 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015949 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015952 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015954 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015957 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:30.019372 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015960 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015963 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015966 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015969 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015972 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015975 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015978 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015981 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015984 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015986 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015989 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015992 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015995 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.015997 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016003 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016006 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016008 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016012 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016014 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016017 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:30.019865 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016019 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:30.020353 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016022 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:30.020353 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016025 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:30.020353 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016028 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:30.020353 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016030 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:30.020353 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016033 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:30.020353 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016036 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:30.020353 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016038 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:30.020353 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016042 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:30.020353 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016044 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:30.020353 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016047 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:30.020353 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016051 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:30.020353 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016055 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:30.020353 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016058 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:30.020353 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016075 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:30.020353 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016078 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:30.020353 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016081 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:30.020353 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016084 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:30.020353 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016086 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016089 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016092 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016095 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016098 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016101 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016103 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016106 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016110 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016113 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016116 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016119 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016122 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016124 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016127 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016129 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016132 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016135 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016137 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:30.020792 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016140 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016143 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016146 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016148 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016152 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016154 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016157 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016159 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016163 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016166 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016168 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016171 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016173 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016176 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016179 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016181 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016184 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016187 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016190 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016192 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016195 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:30.021263 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016198 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016201 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016204 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016206 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016209 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016212 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016214 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016217 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016220 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016222 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016225 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016227 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016230 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016233 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016237 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016240 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016244 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016247 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016251 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:30.021797 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016253 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016256 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016258 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.016261 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017574 2583 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017593 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017603 2583 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017607 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017612 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017616 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017620 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017625 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017628 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017632 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017636 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017639 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017643 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017646 2583 flags.go:64] FLAG: --cgroup-root="" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017649 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017652 2583 flags.go:64] FLAG: --client-ca-file="" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017655 2583 flags.go:64] FLAG: --cloud-provider="external" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017658 2583 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017662 2583 flags.go:64] FLAG: --cluster-domain="" Nov 03 22:06:30.022269 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017666 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017670 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017674 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017677 2583 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017681 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017685 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017688 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017694 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017697 2583 flags.go:64] FLAG: --contention-profiling="false" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017700 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017704 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017707 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017710 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017715 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017718 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017721 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017724 2583 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017728 2583 flags.go:64] FLAG: --enable-server="true" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017731 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017738 2583 flags.go:64] FLAG: --event-burst="100" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017741 2583 flags.go:64] FLAG: --event-qps="50" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017744 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017748 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017751 2583 flags.go:64] FLAG: --eviction-hard="" Nov 03 22:06:30.022804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017756 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017759 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017762 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017766 2583 flags.go:64] FLAG: --eviction-soft="" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017769 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017772 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017775 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017778 2583 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017782 2583 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017785 2583 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017788 2583 flags.go:64] FLAG: --feature-gates="" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017794 2583 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017797 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017801 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017804 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017808 2583 flags.go:64] FLAG: --healthz-port="10248" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017812 2583 flags.go:64] FLAG: --help="false" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017815 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017818 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017822 2583 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017825 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017828 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017832 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 22:06:30.023551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017835 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017838 2583 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017841 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017844 2583 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017847 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017851 2583 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017854 2583 flags.go:64] FLAG: --kube-reserved="" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017857 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017860 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017864 2583 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017867 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017870 2583 flags.go:64] FLAG: --lock-file="" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017873 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017877 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017880 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017887 2583 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017890 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017893 2583 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017896 2583 flags.go:64] FLAG: --logging-format="text" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017900 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017903 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017906 2583 flags.go:64] FLAG: --manifest-url="" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017910 2583 flags.go:64] FLAG: --manifest-url-header="" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017915 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017919 2583 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 22:06:30.024122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017922 2583 flags.go:64] FLAG: --max-pods="110" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017926 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017930 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017933 2583 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017936 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017939 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017942 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017945 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017952 2583 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017956 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017959 2583 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017963 2583 flags.go:64] FLAG: --pod-cidr="" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017966 2583 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017968 2583 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017972 2583 flags.go:64] FLAG: --pods-per-core="0" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017975 2583 flags.go:64] FLAG: --port="10250" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017978 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017981 2583 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-03bb5b8a81e465a71" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017984 2583 flags.go:64] FLAG: --qos-reserved="" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017987 2583 flags.go:64] FLAG: --read-only-port="10255" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017990 2583 flags.go:64] FLAG: --register-node="true" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017993 2583 flags.go:64] FLAG: --register-with-taints="" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.017997 2583 flags.go:64] FLAG: --registry-burst="10" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018003 2583 flags.go:64] FLAG: --registry-qps="5" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018006 2583 flags.go:64] FLAG: --reserved-cpus="" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018009 2583 flags.go:64] FLAG: --reserved-memory="" Nov 03 22:06:30.024849 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018014 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018017 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018021 2583 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018024 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018027 2583 flags.go:64] FLAG: --runonce="false" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018030 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018035 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018038 2583 flags.go:64] FLAG: --seccomp-default="false" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018042 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018045 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018049 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018052 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018055 2583 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018058 2583 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018072 2583 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018075 2583 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018078 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018082 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018085 2583 flags.go:64] FLAG: --system-cgroups="" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018088 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018094 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018097 2583 flags.go:64] FLAG: --tls-cert-file="" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018100 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018105 2583 flags.go:64] FLAG: --tls-min-version="" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018108 2583 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 22:06:30.026037 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018111 2583 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018114 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018118 2583 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018121 2583 flags.go:64] FLAG: --v="2" Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018133 2583 flags.go:64] FLAG: --version="false" Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018138 2583 flags.go:64] FLAG: --vmodule="" Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018142 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018145 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018294 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018298 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018301 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018304 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018307 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018311 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018314 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018318 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018322 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018325 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018328 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018331 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018334 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:30.027135 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018336 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018339 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018342 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018345 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018348 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018350 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018353 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018356 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018359 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018361 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018364 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018367 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018370 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018373 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018376 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018380 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018383 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018385 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018388 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:30.028025 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018391 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018393 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018396 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018399 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018401 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018405 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018408 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018411 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018414 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018417 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018420 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018422 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018425 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018428 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018430 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018433 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018436 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018438 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018441 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:30.028958 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018445 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018449 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018452 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018455 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018457 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018460 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018463 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018465 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018468 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018472 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018475 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018478 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018481 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018483 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018487 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018489 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018492 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018495 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018498 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018501 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:30.029789 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018504 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018507 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018509 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018514 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018519 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018522 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018525 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018528 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018531 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018533 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018536 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018538 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018541 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018544 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018547 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018549 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018552 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018555 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018558 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:30.030382 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018560 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:30.031103 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018563 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:30.031103 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018567 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:30.031103 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018570 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:30.031103 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018572 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:30.031103 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018575 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:30.031103 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018578 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:30.031103 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018581 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:30.031103 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018583 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:30.031103 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018586 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:30.031103 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018589 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:30.031103 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018593 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:30.031103 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018596 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:30.031103 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018598 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:30.031103 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018601 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:30.031103 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018603 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:30.031103 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.018633 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018675 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018681 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018684 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018688 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018691 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018695 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018697 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018700 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018704 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018707 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018710 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018712 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018715 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018718 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018721 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018723 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018726 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018729 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:30.031759 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018733 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018736 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018739 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018742 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018744 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018747 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018750 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018753 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018759 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018761 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018764 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018766 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018769 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018772 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018775 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018778 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018781 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018784 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018787 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:30.032425 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018790 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018792 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018795 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018798 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018801 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018804 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018806 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018809 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018811 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018814 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018817 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018819 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018824 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018831 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018835 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018837 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018840 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018843 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018846 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018848 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:30.032995 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018851 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018855 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018858 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018861 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018864 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018866 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018869 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018873 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018875 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018879 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018881 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018884 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018887 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018890 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018892 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018896 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018900 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018904 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018907 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018909 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:30.033618 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018912 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018915 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018917 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018920 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018922 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018926 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018928 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018931 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018934 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018936 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018939 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018942 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018945 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018949 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018952 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018954 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018957 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018960 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018962 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:30.034225 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018965 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018967 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018970 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018973 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018976 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018978 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018981 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018983 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018987 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.018990 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.027618 2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.027645 2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.027672 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.027680 2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027792 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027804 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027810 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027815 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027819 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:30.034718 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027824 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027829 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027833 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027838 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027842 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027847 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027852 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027856 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027860 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027865 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027872 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027878 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027883 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027888 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027894 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027898 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027910 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027915 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027919 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:30.035241 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027924 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027928 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027933 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027937 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027942 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027947 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027952 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027957 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027961 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027966 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027970 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027974 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027979 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027983 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027987 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027992 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.027996 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028000 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028004 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:30.035796 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028009 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028013 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028017 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028021 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028026 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028029 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028033 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028037 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028041 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028045 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028050 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028055 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028080 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028085 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028089 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028093 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028097 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028102 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028107 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028111 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:30.036562 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028115 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028119 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028124 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028128 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028132 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028136 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028140 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028144 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028148 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028152 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028156 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028160 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028164 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028169 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028173 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028177 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028182 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028186 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028190 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028195 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:30.037403 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028199 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028204 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028208 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028213 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028218 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028222 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028227 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028231 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028236 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028240 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028245 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028249 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028253 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028258 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028262 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028267 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028272 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028276 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028280 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028285 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:30.038377 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028289 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:30.038985 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028293 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:30.038985 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028297 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:30.038985 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.028353 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:06:30.038985 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028623 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:30.038985 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028633 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:30.038985 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028638 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:30.038985 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028643 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:30.038985 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028648 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:30.038985 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028652 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:30.038985 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028657 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:30.038985 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028661 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:30.038985 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028666 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:30.038985 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028672 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:30.038985 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028679 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:30.038985 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028684 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:30.038985 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028690 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028695 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028700 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028705 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028710 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028714 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028719 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028723 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028727 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028731 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028735 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028740 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028744 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028749 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028753 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028757 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028762 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028766 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028770 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028774 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:30.039422 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028778 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028783 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028787 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028792 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028796 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028800 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028804 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028809 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028813 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028817 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028822 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028826 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028830 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028834 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028839 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028844 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028848 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028852 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028856 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:30.039924 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028860 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028865 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028869 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028873 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028878 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028882 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028887 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028891 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028896 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028900 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028905 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028909 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028913 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028917 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028922 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028927 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028931 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028935 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028939 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:30.040473 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028944 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028948 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028953 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028957 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028961 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028965 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028969 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028974 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028977 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028983 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028988 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.028995 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029001 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029006 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029011 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029015 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029019 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029024 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029028 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:30.041217 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029032 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:30.041878 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029037 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:30.041878 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029041 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:30.041878 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029046 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:30.041878 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029050 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:30.041878 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029055 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:30.041878 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029075 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:30.041878 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029080 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:30.041878 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029084 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:30.041878 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029089 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:30.041878 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029093 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:30.041878 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029098 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:30.041878 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029102 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:30.041878 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029107 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:30.041878 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029111 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:30.041878 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029116 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:30.041878 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029120 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:30.042334 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.029167 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:06:30.042334 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029224 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:06:30.042334 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029231 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:06:30.042334 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029239 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:06:30.042334 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029243 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:06:30.042334 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029248 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:06:30.042334 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029253 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:06:30.042334 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029258 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:06:30.042334 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029262 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:06:30.042334 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029267 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:06:30.042334 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029271 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:06:30.042334 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029276 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:06:30.042334 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029280 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:06:30.042334 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029284 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:06:30.042334 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029288 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:06:30.042334 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029292 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029297 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029301 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029306 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029310 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029315 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029319 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029324 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029328 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029334 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029340 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029345 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029349 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029354 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029358 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029363 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029367 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029371 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029375 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029379 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:06:30.042757 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029384 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029388 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029392 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029397 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029402 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029406 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029414 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029419 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029423 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029428 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029432 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029436 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029441 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029445 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029449 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029453 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029457 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029462 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029467 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:06:30.043355 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029471 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029475 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029480 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029484 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029488 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029493 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029497 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029501 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029505 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029510 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029514 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029518 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029522 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029527 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029531 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029536 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029540 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029545 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029550 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:06:30.043799 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029558 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029563 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029567 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029572 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029577 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029581 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029586 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029590 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029595 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029599 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029603 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029608 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029612 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029616 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029623 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029627 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029632 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029636 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029641 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:06:30.044289 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029645 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029649 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029654 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029658 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029665 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029671 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029676 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029682 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029687 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029691 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029696 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029700 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029704 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029711 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:30.029716 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.029806 2583 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.032937 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.033955 2583 server.go:1014] "Starting client certificate rotation" Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.034088 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:06:30.044787 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.034137 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:06:30.045382 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.036284 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:06:30.045382 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.038681 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:06:30.053530 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.053499 2583 log.go:25] "Validated CRI v1 runtime API" Nov 03 22:06:30.057401 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.057376 2583 log.go:25] "Validated CRI v1 image API" Nov 03 22:06:30.058348 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.058317 2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 22:06:30.064008 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.063982 2583 fs.go:135] Filesystem UUIDs: map[246b339f-ec48-4924-91f8-ce7e37d43afc:/dev/nvme0n1p3 3b0bbcb0-28bc-487e-9a34-f9fbc7a19a79:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 22:06:30.064097 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.064006 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 22:06:30.070141 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.069983 2583 manager.go:217] Machine: {Timestamp:2026-11-03 22:06:30.068711028 +0000 UTC m=+0.309980055 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3117520 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d7e2df2a66aeabefad529ebc636cd SystemUUID:ec2d7e2d-f2a6-6aea-befa-d529ebc636cd BootID:723a35da-0579-423d-8c7f-6c86113ef352 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f9:64:30:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f9:64:30:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:04:26:68:56:21 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 22:06:30.070717 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.070704 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 22:06:30.070872 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.070856 2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 22:06:30.073951 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.073901 2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 22:06:30.074222 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.073948 2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-84.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 22:06:30.074283 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.074224 2583 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 22:06:30.074283 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.074234 2583 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 22:06:30.074283 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.074272 2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:06:30.074406 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.074287 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:06:30.075409 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.075394 2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 22:06:30.079748 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.079730 2583 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 22:06:30.079912 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.079901 2583 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 22:06:30.080652 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.080642 2583 kubelet.go:493] "Attempting to sync node with API server" Nov 03 22:06:30.080699 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.080657 2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 22:06:30.080699 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.080670 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 22:06:30.080699 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.080685 2583 kubelet.go:405] "Adding apiserver pod source" Nov 03 22:06:30.080699 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.080694 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 22:06:30.081097 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.081083 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:06:30.083431 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.083413 2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 22:06:30.089512 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.089491 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:06:30.090464 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.090450 2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 22:06:30.090513 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.090482 2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 22:06:30.090655 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.090645 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 22:06:30.090684 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.090660 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 22:06:30.090684 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.090667 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 22:06:30.090684 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.090672 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 22:06:30.090684 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.090677 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 22:06:30.090684 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.090683 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 22:06:30.091413 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.091403 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 22:06:30.091413 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.091413 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 22:06:30.091467 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.091421 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 22:06:30.091467 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.091428 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 22:06:30.091467 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.091438 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 22:06:30.091467 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.091448 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 22:06:30.092126 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.092115 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 22:06:30.092126 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.092126 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 22:06:30.096144 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.096125 2583 server.go:1263] "Started kubelet" Nov 03 22:06:30.096231 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.096182 2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 22:06:30.096322 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.096279 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 22:06:30.096379 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.096345 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 22:06:30.096958 ip-10-0-141-84 systemd[1]: Started Kubernetes Kubelet. Nov 03 22:06:30.097844 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.097827 2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 22:06:30.100750 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.100728 2583 server.go:317] "Adding debug handlers to kubelet server" Nov 03 22:06:30.105228 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:30.105200 2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 22:06:30.111474 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.111453 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 22:06:30.112040 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.112021 2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 22:06:30.112354 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:30.112330 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-84.us-west-2.compute.internal\" not found" Nov 03 22:06:30.112950 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.112931 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 22:06:30.112950 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.112931 2583 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 22:06:30.113124 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.112965 2583 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 22:06:30.113124 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.113047 2583 reconstruct.go:97] "Volume reconstruction finished" Nov 03 22:06:30.113124 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.113057 2583 reconciler.go:29] "Reconciler: start to sync state" Nov 03 22:06:30.113583 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.113566 2583 factory.go:153] Registering CRI-O factory Nov 03 22:06:30.113583 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.113583 2583 factory.go:223] Registration of the crio container factory successfully Nov 03 22:06:30.113730 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.113639 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 22:06:30.113730 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.113651 2583 factory.go:55] Registering systemd factory Nov 03 22:06:30.113730 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.113659 2583 factory.go:223] Registration of the systemd container factory successfully Nov 03 22:06:30.113730 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.113674 2583 factory.go:103] Registering Raw factory Nov 03 22:06:30.113730 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.113687 2583 manager.go:1199] Started watching for new ooms in manager Nov 03 22:06:30.114736 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.114718 2583 manager.go:319] Starting recovery of all containers Nov 03 22:06:30.120483 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.120443 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 22:06:30.121329 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.121313 2583 manager.go:324] Recovery completed Nov 03 22:06:30.125513 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.125486 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:30.128139 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.128052 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:30.128250 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.128155 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:30.128250 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.128171 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:30.128824 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.128807 2583 cpu_manager.go:226] "Starting" policy="none" Nov 03 22:06:30.128824 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.128820 2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 22:06:30.128939 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.128838 2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 22:06:30.128939 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.128872 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:30.129799 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.129782 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:30.129877 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.129815 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:30.129877 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.129828 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:30.131153 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.131081 2583 policy_none.go:50] "Start" Nov 03 22:06:30.131153 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.131101 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 22:06:30.131153 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.131114 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 22:06:30.133135 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.133121 2583 policy_none.go:44] "Start" Nov 03 22:06:30.212944 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:30.212912 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-84.us-west-2.compute.internal\" not found" Nov 03 22:06:30.247098 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:30.247038 2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-84.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 22:06:30.247262 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.247045 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-84.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 22:06:30.249791 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:30.247022 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-84.us-west-2.compute.internal.18e4a84aa1cf34c1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-84.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-84.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 22:06:30.096057537 +0000 UTC m=+0.337326562,LastTimestamp:2026-11-03 22:06:30.096057537 +0000 UTC m=+0.337326562,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-84.us-west-2.compute.internal,}" Nov 03 22:06:30.277081 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.277031 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 22:06:30.277081 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.277075 2583 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 22:06:30.277304 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.277101 2583 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 22:06:30.277304 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:30.277140 2583 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 22:06:30.279804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.279770 2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 22:06:30.282698 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.282679 2583 manager.go:341] "Starting Device Plugin manager" Nov 03 22:06:30.282822 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:30.282717 2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 22:06:30.282822 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.282728 2583 server.go:87] "Starting device plugin registration server" Nov 03 22:06:30.283051 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.283039 2583 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 22:06:30.283116 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.283055 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 22:06:30.283196 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.283178 2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 22:06:30.283283 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.283264 2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 22:06:30.283283 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.283277 2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 22:06:30.283864 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:30.283836 2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 22:06:30.283958 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:30.283885 2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-84.us-west-2.compute.internal\" not found" Nov 03 22:06:30.321754 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.321721 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 22:06:30.321942 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.321785 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5cxmm" Nov 03 22:06:30.377807 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.377752 2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal"] Nov 03 22:06:30.378004 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.377841 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:30.378888 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.378868 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:30.378983 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.378904 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:30.378983 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.378921 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:30.381257 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.381236 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:30.381433 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.381414 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.381486 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.381452 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:30.382101 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.382086 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:30.382205 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.382097 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:30.382205 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.382113 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:30.382205 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.382125 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:30.382205 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.382128 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:30.382205 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.382135 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:30.383961 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.383947 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:30.384279 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.384263 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.384332 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.384298 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:06:30.384836 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.384815 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:30.384919 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.384853 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:30.384919 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.384868 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:30.384919 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.384895 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.385094 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.385027 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:06:30.385094 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.385072 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:06:30.385094 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.385089 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:06:30.394949 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.394923 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:06:30.396442 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.396423 2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:06:30.412255 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:30.412220 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-84.us-west-2.compute.internal\" not found" node="ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.414247 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.414229 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36effd34d282541d8cc7db737926b500-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal\" (UID: \"36effd34d282541d8cc7db737926b500\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.414321 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.414258 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36effd34d282541d8cc7db737926b500-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal\" (UID: \"36effd34d282541d8cc7db737926b500\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.414321 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.414278 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3ff6dbfd235d6871a32270ec8f4d659-config\") pod \"kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal\" (UID: \"a3ff6dbfd235d6871a32270ec8f4d659\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.415179 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:30.415161 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-84.us-west-2.compute.internal\" not found" node="ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.487957 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.487753 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:06:30.488697 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.488676 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:06:30.488865 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.488844 2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.488932 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:30.488876 2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-84.us-west-2.compute.internal\": node \"ip-10-0-141-84.us-west-2.compute.internal\" not found" Nov 03 22:06:30.513496 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.513461 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.514619 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.514592 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36effd34d282541d8cc7db737926b500-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal\" (UID: \"36effd34d282541d8cc7db737926b500\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.514691 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.514628 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36effd34d282541d8cc7db737926b500-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal\" (UID: \"36effd34d282541d8cc7db737926b500\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.514691 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.514649 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3ff6dbfd235d6871a32270ec8f4d659-config\") pod \"kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal\" (UID: \"a3ff6dbfd235d6871a32270ec8f4d659\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.514753 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.514708 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3ff6dbfd235d6871a32270ec8f4d659-config\") pod \"kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal\" (UID: \"a3ff6dbfd235d6871a32270ec8f4d659\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.514753 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.514711 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36effd34d282541d8cc7db737926b500-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal\" (UID: \"36effd34d282541d8cc7db737926b500\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.514753 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.514713 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36effd34d282541d8cc7db737926b500-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal\" (UID: \"36effd34d282541d8cc7db737926b500\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.568551 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.568460 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:06:30.569306 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.569292 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.655638 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.655611 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:06:30.716308 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.716259 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:30.719698 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:30.719674 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:06:31.034486 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.034397 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 22:06:31.034998 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.034573 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 22:06:31.034998 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.034574 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 22:06:31.034998 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.034582 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 22:06:31.034998 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:31.034585 2583 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a3890ba9e34074c9bb3a6df7076aca19-87013095a60c6b5c.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-84.us-west-2.compute.internal.18e4a84aa3b8aad5\": read tcp 10.0.141.84:51366->16.58.171.225:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-84.us-west-2.compute.internal.18e4a84aa3b8aad5 default 4858 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-84.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-84.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-84.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 22:06:30 +0000 UTC,LastTimestamp:2026-11-03 22:06:30.382099073 +0000 UTC m=+0.623368099,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-84.us-west-2.compute.internal,}" Nov 03 22:06:31.081829 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.081800 2583 apiserver.go:52] "Watching apiserver" Nov 03 22:06:31.112315 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.112264 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 22:06:31.327050 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.326940 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:01:30 +0000 UTC" deadline="2028-06-26 08:04:18.932543869 +0000 UTC" Nov 03 22:06:31.327050 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.326982 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14409h57m47.605565007s" Nov 03 22:06:31.375644 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.375606 2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:06:31.375954 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.375929 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-t6b4b","kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal","openshift-dns/node-resolver-79sq5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-2g9vv","kube-system/konnectivity-agent-k4rnh","openshift-multus/network-metrics-daemon-sfqbq","openshift-ovn-kubernetes/ovnkube-node-78tk2","openshift-cluster-node-tuning-operator/tuned-pk5rg","openshift-multus/multus-jl87t","openshift-network-diagnostics/network-check-target-z4ct8","openshift-network-operator/iptables-alerter-gsmpr","kube-system/global-pull-secret-syncer-8wmmt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6"] Nov 03 22:06:31.378639 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.378612 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.380879 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.380855 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:31.381022 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:31.380997 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4ct8" podUID="fab1061b-bf17-477e-bfdf-cb8b1d76b0ff" Nov 03 22:06:31.383229 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.383211 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-79sq5" Nov 03 22:06:31.385547 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.385527 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.387832 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.387806 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k4rnh" Nov 03 22:06:31.390061 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.390038 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.392443 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.392418 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.394798 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.394779 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t6b4b" Nov 03 22:06:31.397368 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.397345 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.399797 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.399757 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gsmpr" Nov 03 22:06:31.402692 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.402669 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:31.402769 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:31.402753 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfqbq" podUID="393f28cb-5fc8-4b3e-8852-eebe41b42a2b" Nov 03 22:06:31.405226 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.405205 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:31.405301 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:31.405271 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wmmt" podUID="99d8e137-a970-441d-9800-d1ce1c8f1f94" Nov 03 22:06:31.413988 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.413960 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 22:06:31.419980 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.419947 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-cni-binary-copy\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.420069 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.419988 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-var-lib-cni-multus\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.420069 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420006 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-var-lib-kubelet\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.420272 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420087 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3e7a098-088b-4e25-ac35-dc531d619b79-cnibin\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.420272 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420125 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-run\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.420272 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420173 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-run-systemd\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.420272 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420226 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-node-log\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.420272 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420254 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-cni-bin\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.420272 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420272 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-cni-netd\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.420471 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420316 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99d8e137-a970-441d-9800-d1ce1c8f1f94-kubelet-config\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:31.420471 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420354 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-cni-dir\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.420471 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420382 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-var-lib-openvswitch\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.420471 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420423 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-run-ovn\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.420607 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420480 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95246059-e5c1-4b42-9678-a1c8b2f63e80-host\") pod \"node-ca-t6b4b\" (UID: \"95246059-e5c1-4b42-9678-a1c8b2f63e80\") " pod="openshift-image-registry/node-ca-t6b4b" Nov 03 22:06:31.420607 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420546 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvmk9\" (UniqueName: \"kubernetes.io/projected/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-kube-api-access-tvmk9\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.420607 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420576 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b3e7a098-088b-4e25-ac35-dc531d619b79-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.420607 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420601 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f4d66c00-b01b-4c14-998a-d84707b4600b-iptables-alerter-script\") pod \"iptables-alerter-gsmpr\" (UID: \"f4d66c00-b01b-4c14-998a-d84707b4600b\") " pod="openshift-network-operator/iptables-alerter-gsmpr" Nov 03 22:06:31.420803 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420626 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-cnibin\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.420803 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420662 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-run-netns\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.420803 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420685 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-etc-kubernetes\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.420803 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420721 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19a326ff-4502-4590-9c0d-194f8cf14abd-konnectivity-ca\") pod \"konnectivity-agent-k4rnh\" (UID: \"19a326ff-4502-4590-9c0d-194f8cf14abd\") " pod="kube-system/konnectivity-agent-k4rnh" Nov 03 22:06:31.420803 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420783 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-systemd\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.421014 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420816 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmppm\" (UniqueName: \"kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm\") pod \"network-check-target-z4ct8\" (UID: \"fab1061b-bf17-477e-bfdf-cb8b1d76b0ff\") " pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:31.421014 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420865 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.421014 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420891 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-run-k8s-cni-cncf-io\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.421014 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420917 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlwbf\" (UniqueName: \"kubernetes.io/projected/b3e7a098-088b-4e25-ac35-dc531d619b79-kube-api-access-vlwbf\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.421014 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420953 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-run-openvswitch\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.421014 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.420985 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d49e0d2-566e-472c-9c2d-c85256d52de4-ovnkube-script-lib\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.421313 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421022 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn75q\" (UniqueName: \"kubernetes.io/projected/f4d66c00-b01b-4c14-998a-d84707b4600b-kube-api-access-fn75q\") pod \"iptables-alerter-gsmpr\" (UID: \"f4d66c00-b01b-4c14-998a-d84707b4600b\") " pod="openshift-network-operator/iptables-alerter-gsmpr" Nov 03 22:06:31.421313 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421041 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/866a1e7a-4728-4bd5-a88e-1a97f36f1eac-hosts-file\") pod \"node-resolver-79sq5\" (UID: \"866a1e7a-4728-4bd5-a88e-1a97f36f1eac\") " pod="openshift-dns/node-resolver-79sq5" Nov 03 22:06:31.421313 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421058 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz5dx\" (UniqueName: \"kubernetes.io/projected/8b599076-85db-4b98-8636-485e964a5e59-kube-api-access-sz5dx\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.421313 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421132 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-kubelet\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.421313 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421182 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-socket-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.421313 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421220 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-etc-selinux\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.421313 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421263 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-var-lib-cni-bin\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.421313 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421291 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-lib-modules\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.421313 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421314 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-slash\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.421587 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421369 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d49e0d2-566e-472c-9c2d-c85256d52de4-ovnkube-config\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.421587 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421402 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d49e0d2-566e-472c-9c2d-c85256d52de4-ovn-node-metrics-cert\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.421587 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421457 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-var-lib-kubelet\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.421587 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421475 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19a326ff-4502-4590-9c0d-194f8cf14abd-agent-certs\") pod \"konnectivity-agent-k4rnh\" (UID: \"19a326ff-4502-4590-9c0d-194f8cf14abd\") " pod="kube-system/konnectivity-agent-k4rnh" Nov 03 22:06:31.421587 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421491 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-host\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.421587 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421532 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b599076-85db-4b98-8636-485e964a5e59-etc-tuned\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.421587 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421566 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95246059-e5c1-4b42-9678-a1c8b2f63e80-serviceca\") pod \"node-ca-t6b4b\" (UID: \"95246059-e5c1-4b42-9678-a1c8b2f63e80\") " pod="openshift-image-registry/node-ca-t6b4b" Nov 03 22:06:31.421790 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421595 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3e7a098-088b-4e25-ac35-dc531d619b79-system-cni-dir\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.421790 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421621 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b3e7a098-088b-4e25-ac35-dc531d619b79-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.421790 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421644 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-daemon-config\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.421790 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421675 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-run-multus-certs\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.421790 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421703 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-kubernetes\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.421790 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421750 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b599076-85db-4b98-8636-485e964a5e59-tmp\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.421790 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421788 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-device-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.421996 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421819 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-hostroot\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.421996 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421844 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3e7a098-088b-4e25-ac35-dc531d619b79-cni-binary-copy\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.421996 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421875 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-sysconfig\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.421996 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421896 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-log-socket\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.421996 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421920 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-run-ovn-kubernetes\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.421996 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421952 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.421996 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421973 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:31.421996 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.421995 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-systemd-units\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.422259 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422021 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-run-netns\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.422259 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422043 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d49e0d2-566e-472c-9c2d-c85256d52de4-env-overrides\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.422259 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422066 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4d66c00-b01b-4c14-998a-d84707b4600b-host-slash\") pod \"iptables-alerter-gsmpr\" (UID: \"f4d66c00-b01b-4c14-998a-d84707b4600b\") " pod="openshift-network-operator/iptables-alerter-gsmpr" Nov 03 22:06:31.422259 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422115 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgxtv\" (UniqueName: \"kubernetes.io/projected/95246059-e5c1-4b42-9678-a1c8b2f63e80-kube-api-access-lgxtv\") pod \"node-ca-t6b4b\" (UID: \"95246059-e5c1-4b42-9678-a1c8b2f63e80\") " pod="openshift-image-registry/node-ca-t6b4b" Nov 03 22:06:31.422259 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422136 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb6r6\" (UniqueName: \"kubernetes.io/projected/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-kube-api-access-vb6r6\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:31.422259 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422153 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-sys\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.422259 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422205 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-sys-fs\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.422259 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422231 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4drgl\" (UniqueName: \"kubernetes.io/projected/412af756-be10-401f-8413-419d66bafaad-kube-api-access-4drgl\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.422259 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422250 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:31.422599 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422291 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-conf-dir\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.422599 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422336 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm7gx\" (UniqueName: \"kubernetes.io/projected/866a1e7a-4728-4bd5-a88e-1a97f36f1eac-kube-api-access-sm7gx\") pod \"node-resolver-79sq5\" (UID: \"866a1e7a-4728-4bd5-a88e-1a97f36f1eac\") " pod="openshift-dns/node-resolver-79sq5" Nov 03 22:06:31.422599 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422367 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3e7a098-088b-4e25-ac35-dc531d619b79-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.422599 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422411 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-sysctl-d\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.422599 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422440 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-sysctl-conf\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.422599 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422477 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vt2f\" (UniqueName: \"kubernetes.io/projected/1d49e0d2-566e-472c-9c2d-c85256d52de4-kube-api-access-6vt2f\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.422599 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422540 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-socket-dir-parent\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.422599 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422569 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866a1e7a-4728-4bd5-a88e-1a97f36f1eac-tmp-dir\") pod \"node-resolver-79sq5\" (UID: \"866a1e7a-4728-4bd5-a88e-1a97f36f1eac\") " pod="openshift-dns/node-resolver-79sq5" Nov 03 22:06:31.422599 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422596 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-modprobe-d\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.422974 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422632 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-etc-openvswitch\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.422974 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422663 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-registration-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.422974 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422690 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99d8e137-a970-441d-9800-d1ce1c8f1f94-dbus\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:31.422974 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.422716 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-system-cni-dir\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.436275 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.436246 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:06:31.436476 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.436320 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:06:31.436476 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.436337 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:06:31.436476 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.436425 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:06:31.436476 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.436425 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bkplg\"" Nov 03 22:06:31.440387 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.440361 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:06:31.440480 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.440394 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:06:31.440480 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.440450 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ckrtj\"" Nov 03 22:06:31.442356 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.442338 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:06:31.442466 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.442453 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dngqk\"" Nov 03 22:06:31.442525 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.442465 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:06:31.447124 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.445069 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:06:31.447124 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.445151 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m27qh\"" Nov 03 22:06:31.447124 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.445487 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:06:31.447124 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.447076 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:06:31.447434 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.447248 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dgskz\"" Nov 03 22:06:31.447434 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.447269 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:06:31.449267 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.449227 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:06:31.449267 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.449228 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 22:06:31.449434 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.449292 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j54w4\"" Nov 03 22:06:31.449434 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.449347 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:06:31.523582 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523550 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-daemon-config\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.523745 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523588 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-run-multus-certs\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.523745 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523617 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-kubernetes\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.523745 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523681 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-run-multus-certs\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.523926 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523743 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b599076-85db-4b98-8636-485e964a5e59-tmp\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.523926 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523778 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-device-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.523926 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523750 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-kubernetes\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.523926 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523803 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-hostroot\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.523926 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523830 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3e7a098-088b-4e25-ac35-dc531d619b79-cni-binary-copy\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.523926 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523857 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-sysconfig\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.523926 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523871 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-hostroot\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.523926 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523882 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-log-socket\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.523926 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523894 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-device-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.523926 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523908 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-run-ovn-kubernetes\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.524445 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523938 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-sysconfig\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.524445 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523948 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-run-ovn-kubernetes\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.524445 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523962 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.524445 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523983 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-log-socket\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.524445 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.523994 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:31.524445 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524014 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.524445 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524026 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-systemd-units\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.524445 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524105 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-run-netns\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.524445 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524113 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-systemd-units\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.524445 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524132 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d49e0d2-566e-472c-9c2d-c85256d52de4-env-overrides\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.524445 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:31.524148 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:31.524445 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524160 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-run-netns\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.524445 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524159 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4d66c00-b01b-4c14-998a-d84707b4600b-host-slash\") pod \"iptables-alerter-gsmpr\" (UID: \"f4d66c00-b01b-4c14-998a-d84707b4600b\") " pod="openshift-network-operator/iptables-alerter-gsmpr" Nov 03 22:06:31.524445 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524180 2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 22:06:31.524445 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524201 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgxtv\" (UniqueName: \"kubernetes.io/projected/95246059-e5c1-4b42-9678-a1c8b2f63e80-kube-api-access-lgxtv\") pod \"node-ca-t6b4b\" (UID: \"95246059-e5c1-4b42-9678-a1c8b2f63e80\") " pod="openshift-image-registry/node-ca-t6b4b" Nov 03 22:06:31.524445 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:31.524237 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret podName:99d8e137-a970-441d-9800-d1ce1c8f1f94 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:32.024213068 +0000 UTC m=+2.265482097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret") pod "global-pull-secret-syncer-8wmmt" (UID: "99d8e137-a970-441d-9800-d1ce1c8f1f94") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:31.524445 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524242 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4d66c00-b01b-4c14-998a-d84707b4600b-host-slash\") pod \"iptables-alerter-gsmpr\" (UID: \"f4d66c00-b01b-4c14-998a-d84707b4600b\") " pod="openshift-network-operator/iptables-alerter-gsmpr" Nov 03 22:06:31.525241 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524265 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-daemon-config\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.525241 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524254 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb6r6\" (UniqueName: \"kubernetes.io/projected/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-kube-api-access-vb6r6\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:31.525241 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524331 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-sys\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.525241 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524358 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-sys-fs\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.525241 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524384 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drgl\" (UniqueName: \"kubernetes.io/projected/412af756-be10-401f-8413-419d66bafaad-kube-api-access-4drgl\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.525241 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524409 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:31.525241 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524433 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-conf-dir\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.525241 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524408 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3e7a098-088b-4e25-ac35-dc531d619b79-cni-binary-copy\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.525241 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524447 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-sys\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.525241 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524460 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm7gx\" (UniqueName: \"kubernetes.io/projected/866a1e7a-4728-4bd5-a88e-1a97f36f1eac-kube-api-access-sm7gx\") pod \"node-resolver-79sq5\" (UID: \"866a1e7a-4728-4bd5-a88e-1a97f36f1eac\") " pod="openshift-dns/node-resolver-79sq5" Nov 03 22:06:31.525241 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524432 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-sys-fs\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.525241 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:31.524497 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:31.525241 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524510 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-conf-dir\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.525241 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524496 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3e7a098-088b-4e25-ac35-dc531d619b79-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.525241 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:31.524543 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs podName:393f28cb-5fc8-4b3e-8852-eebe41b42a2b nodeName:}" failed. No retries permitted until 2026-11-03 22:06:32.024526953 +0000 UTC m=+2.265795984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs") pod "network-metrics-daemon-sfqbq" (UID: "393f28cb-5fc8-4b3e-8852-eebe41b42a2b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:31.525241 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524583 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-sysctl-d\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.525241 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524609 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-sysctl-conf\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.525978 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524603 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3e7a098-088b-4e25-ac35-dc531d619b79-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.525978 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524634 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vt2f\" (UniqueName: \"kubernetes.io/projected/1d49e0d2-566e-472c-9c2d-c85256d52de4-kube-api-access-6vt2f\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.525978 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524665 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-socket-dir-parent\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.525978 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524692 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866a1e7a-4728-4bd5-a88e-1a97f36f1eac-tmp-dir\") pod \"node-resolver-79sq5\" (UID: \"866a1e7a-4728-4bd5-a88e-1a97f36f1eac\") " pod="openshift-dns/node-resolver-79sq5" Nov 03 22:06:31.525978 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524709 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-sysctl-d\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.525978 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524720 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-modprobe-d\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.525978 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524746 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-etc-openvswitch\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.525978 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524770 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-registration-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.525978 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524793 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-sysctl-conf\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.525978 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524802 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-socket-dir-parent\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.525978 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524795 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99d8e137-a970-441d-9800-d1ce1c8f1f94-dbus\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:31.525978 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524846 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-system-cni-dir\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.525978 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524860 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99d8e137-a970-441d-9800-d1ce1c8f1f94-dbus\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:31.525978 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524873 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-cni-binary-copy\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.525978 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524883 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-registration-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.525978 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524899 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-var-lib-cni-multus\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.525978 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524929 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-var-lib-kubelet\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.526875 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524945 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-system-cni-dir\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.526875 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524955 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3e7a098-088b-4e25-ac35-dc531d619b79-cnibin\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.526875 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524964 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-etc-openvswitch\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.526875 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524932 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-modprobe-d\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.526875 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524980 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-run\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.526875 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.524996 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-var-lib-kubelet\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.526875 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525013 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-run-systemd\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.526875 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525023 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-run\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.526875 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525028 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866a1e7a-4728-4bd5-a88e-1a97f36f1eac-tmp-dir\") pod \"node-resolver-79sq5\" (UID: \"866a1e7a-4728-4bd5-a88e-1a97f36f1eac\") " pod="openshift-dns/node-resolver-79sq5" Nov 03 22:06:31.526875 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525039 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-node-log\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.526875 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525064 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3e7a098-088b-4e25-ac35-dc531d619b79-cnibin\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.526875 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525037 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-var-lib-cni-multus\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.526875 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525082 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-cni-bin\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.526875 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525090 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-run-systemd\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.526875 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525121 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-node-log\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.526875 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525107 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-cni-netd\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.526875 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525139 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-cni-bin\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.526875 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525160 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-cni-netd\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.527805 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525177 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99d8e137-a970-441d-9800-d1ce1c8f1f94-kubelet-config\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:31.527805 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525197 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-cni-dir\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.527805 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525213 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-var-lib-openvswitch\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.527805 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525228 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-run-ovn\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.527805 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525241 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99d8e137-a970-441d-9800-d1ce1c8f1f94-kubelet-config\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:31.527805 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525243 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95246059-e5c1-4b42-9678-a1c8b2f63e80-host\") pod \"node-ca-t6b4b\" (UID: \"95246059-e5c1-4b42-9678-a1c8b2f63e80\") " pod="openshift-image-registry/node-ca-t6b4b" Nov 03 22:06:31.527805 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525286 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95246059-e5c1-4b42-9678-a1c8b2f63e80-host\") pod \"node-ca-t6b4b\" (UID: \"95246059-e5c1-4b42-9678-a1c8b2f63e80\") " pod="openshift-image-registry/node-ca-t6b4b" Nov 03 22:06:31.527805 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525291 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvmk9\" (UniqueName: \"kubernetes.io/projected/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-kube-api-access-tvmk9\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.527805 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525315 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-cni-dir\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.527805 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525319 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b3e7a098-088b-4e25-ac35-dc531d619b79-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.527805 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525374 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-run-ovn\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.527805 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525350 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-var-lib-openvswitch\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.527805 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525388 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-cni-binary-copy\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.527805 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525376 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f4d66c00-b01b-4c14-998a-d84707b4600b-iptables-alerter-script\") pod \"iptables-alerter-gsmpr\" (UID: \"f4d66c00-b01b-4c14-998a-d84707b4600b\") " pod="openshift-network-operator/iptables-alerter-gsmpr" Nov 03 22:06:31.527805 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525426 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-cnibin\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.527805 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525453 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-run-netns\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.527805 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525470 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-cnibin\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.527805 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525479 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-etc-kubernetes\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.528737 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525510 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19a326ff-4502-4590-9c0d-194f8cf14abd-konnectivity-ca\") pod \"konnectivity-agent-k4rnh\" (UID: \"19a326ff-4502-4590-9c0d-194f8cf14abd\") " pod="kube-system/konnectivity-agent-k4rnh" Nov 03 22:06:31.528737 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525511 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-run-netns\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.528737 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525538 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-systemd\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.528737 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525542 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-etc-kubernetes\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.528737 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525593 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-systemd\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.528737 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525624 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmppm\" (UniqueName: \"kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm\") pod \"network-check-target-z4ct8\" (UID: \"fab1061b-bf17-477e-bfdf-cb8b1d76b0ff\") " pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:31.528737 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525652 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.528737 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525681 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-run-k8s-cni-cncf-io\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.528737 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525711 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlwbf\" (UniqueName: \"kubernetes.io/projected/b3e7a098-088b-4e25-ac35-dc531d619b79-kube-api-access-vlwbf\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.528737 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525731 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.528737 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525742 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-run-openvswitch\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.528737 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525770 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d49e0d2-566e-472c-9c2d-c85256d52de4-ovnkube-script-lib\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.528737 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525780 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-run-k8s-cni-cncf-io\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.528737 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525797 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn75q\" (UniqueName: \"kubernetes.io/projected/f4d66c00-b01b-4c14-998a-d84707b4600b-kube-api-access-fn75q\") pod \"iptables-alerter-gsmpr\" (UID: \"f4d66c00-b01b-4c14-998a-d84707b4600b\") " pod="openshift-network-operator/iptables-alerter-gsmpr" Nov 03 22:06:31.528737 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525816 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b3e7a098-088b-4e25-ac35-dc531d619b79-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.528737 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525825 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/866a1e7a-4728-4bd5-a88e-1a97f36f1eac-hosts-file\") pod \"node-resolver-79sq5\" (UID: \"866a1e7a-4728-4bd5-a88e-1a97f36f1eac\") " pod="openshift-dns/node-resolver-79sq5" Nov 03 22:06:31.528737 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525837 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-run-openvswitch\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.529254 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525852 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz5dx\" (UniqueName: \"kubernetes.io/projected/8b599076-85db-4b98-8636-485e964a5e59-kube-api-access-sz5dx\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.529254 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525903 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-kubelet\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.529254 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525929 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-socket-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.529254 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525928 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/866a1e7a-4728-4bd5-a88e-1a97f36f1eac-hosts-file\") pod \"node-resolver-79sq5\" (UID: \"866a1e7a-4728-4bd5-a88e-1a97f36f1eac\") " pod="openshift-dns/node-resolver-79sq5" Nov 03 22:06:31.529254 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525955 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19a326ff-4502-4590-9c0d-194f8cf14abd-konnectivity-ca\") pod \"konnectivity-agent-k4rnh\" (UID: \"19a326ff-4502-4590-9c0d-194f8cf14abd\") " pod="kube-system/konnectivity-agent-k4rnh" Nov 03 22:06:31.529254 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.525958 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-etc-selinux\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.529254 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526008 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-var-lib-cni-bin\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.529254 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526036 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-lib-modules\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.529254 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526086 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-var-lib-cni-bin\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.529254 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526092 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-socket-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.529254 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526117 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-slash\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.529254 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526143 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d49e0d2-566e-472c-9c2d-c85256d52de4-ovnkube-config\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.529254 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526168 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d49e0d2-566e-472c-9c2d-c85256d52de4-ovn-node-metrics-cert\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.529254 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526180 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-slash\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.529254 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526178 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-kubelet\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.529254 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526193 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-var-lib-kubelet\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.529254 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526180 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-etc-selinux\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.529738 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526145 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-lib-modules\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.529738 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526228 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19a326ff-4502-4590-9c0d-194f8cf14abd-agent-certs\") pod \"konnectivity-agent-k4rnh\" (UID: \"19a326ff-4502-4590-9c0d-194f8cf14abd\") " pod="kube-system/konnectivity-agent-k4rnh" Nov 03 22:06:31.529738 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526232 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-var-lib-kubelet\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.529738 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526269 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-host\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.529738 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526371 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b599076-85db-4b98-8636-485e964a5e59-etc-tuned\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.529738 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526390 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95246059-e5c1-4b42-9678-a1c8b2f63e80-serviceca\") pod \"node-ca-t6b4b\" (UID: \"95246059-e5c1-4b42-9678-a1c8b2f63e80\") " pod="openshift-image-registry/node-ca-t6b4b" Nov 03 22:06:31.529738 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526408 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3e7a098-088b-4e25-ac35-dc531d619b79-system-cni-dir\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.529738 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526433 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b3e7a098-088b-4e25-ac35-dc531d619b79-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.529738 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526513 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-host\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.529738 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526520 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3e7a098-088b-4e25-ac35-dc531d619b79-system-cni-dir\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.529738 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.526868 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b3e7a098-088b-4e25-ac35-dc531d619b79-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.529738 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.528597 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b599076-85db-4b98-8636-485e964a5e59-etc-tuned\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.529738 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.528697 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b599076-85db-4b98-8636-485e964a5e59-tmp\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.529738 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.528933 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19a326ff-4502-4590-9c0d-194f8cf14abd-agent-certs\") pod \"konnectivity-agent-k4rnh\" (UID: \"19a326ff-4502-4590-9c0d-194f8cf14abd\") " pod="kube-system/konnectivity-agent-k4rnh" Nov 03 22:06:31.542121 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.542091 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 22:06:31.542268 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.542185 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8sg5q" Nov 03 22:06:31.544822 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.544800 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:06:31.544935 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.544917 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:06:31.545109 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.545089 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:06:31.545534 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.545515 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:06:31.545534 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.545523 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:06:31.545671 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.545528 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:06:31.545671 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.545543 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:06:31.545671 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.545552 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:06:31.545671 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.545565 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:06:31.545671 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.545573 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:06:31.545671 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.545567 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:06:31.545671 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.545639 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8pftz\"" Nov 03 22:06:31.545876 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.545754 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z5g9w\"" Nov 03 22:06:31.545876 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.545754 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5mdhj\"" Nov 03 22:06:31.545876 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.545788 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:06:31.547279 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.547256 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95246059-e5c1-4b42-9678-a1c8b2f63e80-serviceca\") pod \"node-ca-t6b4b\" (UID: \"95246059-e5c1-4b42-9678-a1c8b2f63e80\") " pod="openshift-image-registry/node-ca-t6b4b" Nov 03 22:06:31.547363 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.547306 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d49e0d2-566e-472c-9c2d-c85256d52de4-ovnkube-script-lib\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.547363 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.547312 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d49e0d2-566e-472c-9c2d-c85256d52de4-ovnkube-config\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.549049 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.549028 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d49e0d2-566e-472c-9c2d-c85256d52de4-ovn-node-metrics-cert\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.556000 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.555972 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d49e0d2-566e-472c-9c2d-c85256d52de4-env-overrides\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.556488 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.556470 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f4d66c00-b01b-4c14-998a-d84707b4600b-iptables-alerter-script\") pod \"iptables-alerter-gsmpr\" (UID: \"f4d66c00-b01b-4c14-998a-d84707b4600b\") " pod="openshift-network-operator/iptables-alerter-gsmpr" Nov 03 22:06:31.586376 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:31.586306 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:06:31.586376 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:31.586345 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:06:31.586376 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:31.586361 2583 projected.go:196] Error preparing data for projected volume kube-api-access-jmppm for pod openshift-network-diagnostics/network-check-target-z4ct8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:31.586613 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:31.586443 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm podName:fab1061b-bf17-477e-bfdf-cb8b1d76b0ff nodeName:}" failed. No retries permitted until 2026-11-03 22:06:32.086416132 +0000 UTC m=+2.327685168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jmppm" (UniqueName: "kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm") pod "network-check-target-z4ct8" (UID: "fab1061b-bf17-477e-bfdf-cb8b1d76b0ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:31.588669 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.588635 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm7gx\" (UniqueName: \"kubernetes.io/projected/866a1e7a-4728-4bd5-a88e-1a97f36f1eac-kube-api-access-sm7gx\") pod \"node-resolver-79sq5\" (UID: \"866a1e7a-4728-4bd5-a88e-1a97f36f1eac\") " pod="openshift-dns/node-resolver-79sq5" Nov 03 22:06:31.588804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.588690 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgxtv\" (UniqueName: \"kubernetes.io/projected/95246059-e5c1-4b42-9678-a1c8b2f63e80-kube-api-access-lgxtv\") pod \"node-ca-t6b4b\" (UID: \"95246059-e5c1-4b42-9678-a1c8b2f63e80\") " pod="openshift-image-registry/node-ca-t6b4b" Nov 03 22:06:31.588804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.588635 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvmk9\" (UniqueName: \"kubernetes.io/projected/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-kube-api-access-tvmk9\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.588804 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.588691 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz5dx\" (UniqueName: \"kubernetes.io/projected/8b599076-85db-4b98-8636-485e964a5e59-kube-api-access-sz5dx\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.588997 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.588855 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4drgl\" (UniqueName: \"kubernetes.io/projected/412af756-be10-401f-8413-419d66bafaad-kube-api-access-4drgl\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.589515 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.589486 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn75q\" (UniqueName: \"kubernetes.io/projected/f4d66c00-b01b-4c14-998a-d84707b4600b-kube-api-access-fn75q\") pod \"iptables-alerter-gsmpr\" (UID: \"f4d66c00-b01b-4c14-998a-d84707b4600b\") " pod="openshift-network-operator/iptables-alerter-gsmpr" Nov 03 22:06:31.589797 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.589775 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb6r6\" (UniqueName: \"kubernetes.io/projected/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-kube-api-access-vb6r6\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:31.590546 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.590522 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlwbf\" (UniqueName: \"kubernetes.io/projected/b3e7a098-088b-4e25-ac35-dc531d619b79-kube-api-access-vlwbf\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.590712 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.590695 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vt2f\" (UniqueName: \"kubernetes.io/projected/1d49e0d2-566e-472c-9c2d-c85256d52de4-kube-api-access-6vt2f\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.713790 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.713756 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jl87t" Nov 03 22:06:31.730224 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.730189 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-79sq5" Nov 03 22:06:31.751908 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.751857 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:06:31.765768 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.765725 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k4rnh" Nov 03 22:06:31.768095 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.768032 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:06:31.788058 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.788018 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:06:31.801736 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.801703 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t6b4b" Nov 03 22:06:31.831491 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.831452 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gsmpr" Nov 03 22:06:31.838879 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.838790 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:31.936884 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:31.936847 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:06:32.031647 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:32.031612 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:32.031895 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:32.031655 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:32.031895 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:32.031778 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:32.031895 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:32.031839 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs podName:393f28cb-5fc8-4b3e-8852-eebe41b42a2b nodeName:}" failed. No retries permitted until 2026-11-03 22:06:33.031825909 +0000 UTC m=+3.273094924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs") pod "network-metrics-daemon-sfqbq" (UID: "393f28cb-5fc8-4b3e-8852-eebe41b42a2b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:32.031895 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:32.031779 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:32.031895 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:32.031888 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret podName:99d8e137-a970-441d-9800-d1ce1c8f1f94 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:33.031878089 +0000 UTC m=+3.273147108 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret") pod "global-pull-secret-syncer-8wmmt" (UID: "99d8e137-a970-441d-9800-d1ce1c8f1f94") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:32.099953 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:32.099873 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:06:32.132887 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:32.132846 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmppm\" (UniqueName: \"kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm\") pod \"network-check-target-z4ct8\" (UID: \"fab1061b-bf17-477e-bfdf-cb8b1d76b0ff\") " pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:32.133053 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:32.133000 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:06:32.133053 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:32.133016 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:06:32.133053 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:32.133026 2583 projected.go:196] Error preparing data for projected volume kube-api-access-jmppm for pod openshift-network-diagnostics/network-check-target-z4ct8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:32.133221 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:32.133078 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm podName:fab1061b-bf17-477e-bfdf-cb8b1d76b0ff nodeName:}" failed. No retries permitted until 2026-11-03 22:06:33.133065777 +0000 UTC m=+3.374334791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jmppm" (UniqueName: "kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm") pod "network-check-target-z4ct8" (UID: "fab1061b-bf17-477e-bfdf-cb8b1d76b0ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:32.148804 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:32.148758 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36effd34d282541d8cc7db737926b500.slice/crio-a614e692f58d8f7e0cf3ec28a1e79ac57d2e7bcb798b87460a03fbfc01c9ca56 WatchSource:0}: Error finding container a614e692f58d8f7e0cf3ec28a1e79ac57d2e7bcb798b87460a03fbfc01c9ca56: Status 404 returned error can't find the container with id a614e692f58d8f7e0cf3ec28a1e79ac57d2e7bcb798b87460a03fbfc01c9ca56 Nov 03 22:06:32.149397 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:32.149370 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ff6dbfd235d6871a32270ec8f4d659.slice/crio-a8218e7160e06cba3868bb17fb9f2219dbb98163d029a279ee76a91ec027ee86 WatchSource:0}: Error finding container a8218e7160e06cba3868bb17fb9f2219dbb98163d029a279ee76a91ec027ee86: Status 404 returned error can't find the container with id a8218e7160e06cba3868bb17fb9f2219dbb98163d029a279ee76a91ec027ee86 Nov 03 22:06:32.154837 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:32.154747 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:06:32.281940 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:32.281879 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" event={"ID":"36effd34d282541d8cc7db737926b500","Type":"ContainerStarted","Data":"a614e692f58d8f7e0cf3ec28a1e79ac57d2e7bcb798b87460a03fbfc01c9ca56"} Nov 03 22:06:32.282923 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:32.282898 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal" event={"ID":"a3ff6dbfd235d6871a32270ec8f4d659","Type":"ContainerStarted","Data":"a8218e7160e06cba3868bb17fb9f2219dbb98163d029a279ee76a91ec027ee86"} Nov 03 22:06:32.385256 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:32.385163 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:06:32.545721 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:32.545635 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:01:31 +0000 UTC" deadline="2028-06-24 01:02:22.813216034 +0000 UTC" Nov 03 22:06:32.545721 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:32.545710 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14354h55m50.267508854s" Nov 03 22:06:32.613708 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:32.613677 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b1fdc9_35d8_4bd3_af0c_cea9299192e8.slice/crio-353f4193ce0d8e6456ba82f832354c4d42ef16743649b886194c732eb09d110b WatchSource:0}: Error finding container 353f4193ce0d8e6456ba82f832354c4d42ef16743649b886194c732eb09d110b: Status 404 returned error can't find the container with id 353f4193ce0d8e6456ba82f832354c4d42ef16743649b886194c732eb09d110b Nov 03 22:06:32.717795 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:32.717763 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b599076_85db_4b98_8636_485e964a5e59.slice/crio-f5dc2d21ce5aaa698b73fe1b152adcbf3c0285d84aa04ac65839115eb82e7f99 WatchSource:0}: Error finding container f5dc2d21ce5aaa698b73fe1b152adcbf3c0285d84aa04ac65839115eb82e7f99: Status 404 returned error can't find the container with id f5dc2d21ce5aaa698b73fe1b152adcbf3c0285d84aa04ac65839115eb82e7f99 Nov 03 22:06:32.730941 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:32.730903 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d66c00_b01b_4c14_998a_d84707b4600b.slice/crio-eee432b62ea81e4f68471f2eee8cf21730bd9548455992a6d708e65938d2cc12 WatchSource:0}: Error finding container eee432b62ea81e4f68471f2eee8cf21730bd9548455992a6d708e65938d2cc12: Status 404 returned error can't find the container with id eee432b62ea81e4f68471f2eee8cf21730bd9548455992a6d708e65938d2cc12 Nov 03 22:06:32.735646 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:32.735614 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866a1e7a_4728_4bd5_a88e_1a97f36f1eac.slice/crio-56546b9723ade1aa7fb25fcdd92c1ba84e66ffbdad14ab5738c52195173ff5b5 WatchSource:0}: Error finding container 56546b9723ade1aa7fb25fcdd92c1ba84e66ffbdad14ab5738c52195173ff5b5: Status 404 returned error can't find the container with id 56546b9723ade1aa7fb25fcdd92c1ba84e66ffbdad14ab5738c52195173ff5b5 Nov 03 22:06:32.736787 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:32.736763 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412af756_be10_401f_8413_419d66bafaad.slice/crio-13b009c03f31dd8d6e367b6e482b641a88b43eea787167db3e95fec2fe57e5bb WatchSource:0}: Error finding container 13b009c03f31dd8d6e367b6e482b641a88b43eea787167db3e95fec2fe57e5bb: Status 404 returned error can't find the container with id 13b009c03f31dd8d6e367b6e482b641a88b43eea787167db3e95fec2fe57e5bb Nov 03 22:06:32.780681 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:32.780647 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a326ff_4502_4590_9c0d_194f8cf14abd.slice/crio-1863b3f07d31ef234ca436008cf74eede29968bca9f058fabab4bb6b13679c63 WatchSource:0}: Error finding container 1863b3f07d31ef234ca436008cf74eede29968bca9f058fabab4bb6b13679c63: Status 404 returned error can't find the container with id 1863b3f07d31ef234ca436008cf74eede29968bca9f058fabab4bb6b13679c63 Nov 03 22:06:32.788400 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:32.788368 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e7a098_088b_4e25_ac35_dc531d619b79.slice/crio-b7df88dc34097524a925047e9118094cbb03377f7ad56e9ef8490c94fd865425 WatchSource:0}: Error finding container b7df88dc34097524a925047e9118094cbb03377f7ad56e9ef8490c94fd865425: Status 404 returned error can't find the container with id b7df88dc34097524a925047e9118094cbb03377f7ad56e9ef8490c94fd865425 Nov 03 22:06:32.789600 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:32.789574 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d49e0d2_566e_472c_9c2d_c85256d52de4.slice/crio-aedd4fb1ad6446f67ad8b58eccfa93ee65feb8805918783b757bbf5e8ef6244e WatchSource:0}: Error finding container aedd4fb1ad6446f67ad8b58eccfa93ee65feb8805918783b757bbf5e8ef6244e: Status 404 returned error can't find the container with id aedd4fb1ad6446f67ad8b58eccfa93ee65feb8805918783b757bbf5e8ef6244e Nov 03 22:06:32.828940 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:06:32.828909 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95246059_e5c1_4b42_9678_a1c8b2f63e80.slice/crio-f2bf980b365434d4e4300fb48c5a170bf5982658d90e2c282949e5c638669e08 WatchSource:0}: Error finding container f2bf980b365434d4e4300fb48c5a170bf5982658d90e2c282949e5c638669e08: Status 404 returned error can't find the container with id f2bf980b365434d4e4300fb48c5a170bf5982658d90e2c282949e5c638669e08 Nov 03 22:06:33.039484 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:33.039393 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:33.039484 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:33.039459 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:33.039669 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:33.039546 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:33.039669 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:33.039598 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret podName:99d8e137-a970-441d-9800-d1ce1c8f1f94 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:35.039586337 +0000 UTC m=+5.280855351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret") pod "global-pull-secret-syncer-8wmmt" (UID: "99d8e137-a970-441d-9800-d1ce1c8f1f94") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:33.039669 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:33.039546 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:33.039669 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:33.039637 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs podName:393f28cb-5fc8-4b3e-8852-eebe41b42a2b nodeName:}" failed. No retries permitted until 2026-11-03 22:06:35.03962736 +0000 UTC m=+5.280896375 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs") pod "network-metrics-daemon-sfqbq" (UID: "393f28cb-5fc8-4b3e-8852-eebe41b42a2b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:33.140320 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:33.140281 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmppm\" (UniqueName: \"kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm\") pod \"network-check-target-z4ct8\" (UID: \"fab1061b-bf17-477e-bfdf-cb8b1d76b0ff\") " pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:33.140831 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:33.140395 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:06:33.140831 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:33.140411 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:06:33.140831 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:33.140420 2583 projected.go:196] Error preparing data for projected volume kube-api-access-jmppm for pod openshift-network-diagnostics/network-check-target-z4ct8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:33.140831 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:33.140468 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm podName:fab1061b-bf17-477e-bfdf-cb8b1d76b0ff nodeName:}" failed. No retries permitted until 2026-11-03 22:06:35.140454244 +0000 UTC m=+5.381723257 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jmppm" (UniqueName: "kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm") pod "network-check-target-z4ct8" (UID: "fab1061b-bf17-477e-bfdf-cb8b1d76b0ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:33.277924 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:33.277890 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:33.278147 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:33.277951 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:33.278147 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:33.277961 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:33.278147 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:33.278056 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4ct8" podUID="fab1061b-bf17-477e-bfdf-cb8b1d76b0ff" Nov 03 22:06:33.278321 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:33.278155 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfqbq" podUID="393f28cb-5fc8-4b3e-8852-eebe41b42a2b" Nov 03 22:06:33.278321 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:33.278235 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wmmt" podUID="99d8e137-a970-441d-9800-d1ce1c8f1f94" Nov 03 22:06:33.286736 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:33.286696 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t6b4b" event={"ID":"95246059-e5c1-4b42-9678-a1c8b2f63e80","Type":"ContainerStarted","Data":"f2bf980b365434d4e4300fb48c5a170bf5982658d90e2c282949e5c638669e08"} Nov 03 22:06:33.291473 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:33.291386 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" event={"ID":"1d49e0d2-566e-472c-9c2d-c85256d52de4","Type":"ContainerStarted","Data":"aedd4fb1ad6446f67ad8b58eccfa93ee65feb8805918783b757bbf5e8ef6244e"} Nov 03 22:06:33.292866 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:33.292831 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k4rnh" event={"ID":"19a326ff-4502-4590-9c0d-194f8cf14abd","Type":"ContainerStarted","Data":"1863b3f07d31ef234ca436008cf74eede29968bca9f058fabab4bb6b13679c63"} Nov 03 22:06:33.294211 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:33.294178 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" event={"ID":"412af756-be10-401f-8413-419d66bafaad","Type":"ContainerStarted","Data":"13b009c03f31dd8d6e367b6e482b641a88b43eea787167db3e95fec2fe57e5bb"} Nov 03 22:06:33.295261 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:33.295237 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" event={"ID":"8b599076-85db-4b98-8636-485e964a5e59","Type":"ContainerStarted","Data":"f5dc2d21ce5aaa698b73fe1b152adcbf3c0285d84aa04ac65839115eb82e7f99"} Nov 03 22:06:33.296405 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:33.296380 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jl87t" event={"ID":"37b1fdc9-35d8-4bd3-af0c-cea9299192e8","Type":"ContainerStarted","Data":"353f4193ce0d8e6456ba82f832354c4d42ef16743649b886194c732eb09d110b"} Nov 03 22:06:33.297751 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:33.297722 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2g9vv" event={"ID":"b3e7a098-088b-4e25-ac35-dc531d619b79","Type":"ContainerStarted","Data":"b7df88dc34097524a925047e9118094cbb03377f7ad56e9ef8490c94fd865425"} Nov 03 22:06:33.298965 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:33.298938 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-79sq5" event={"ID":"866a1e7a-4728-4bd5-a88e-1a97f36f1eac","Type":"ContainerStarted","Data":"56546b9723ade1aa7fb25fcdd92c1ba84e66ffbdad14ab5738c52195173ff5b5"} Nov 03 22:06:33.300149 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:33.300124 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gsmpr" event={"ID":"f4d66c00-b01b-4c14-998a-d84707b4600b","Type":"ContainerStarted","Data":"eee432b62ea81e4f68471f2eee8cf21730bd9548455992a6d708e65938d2cc12"} Nov 03 22:06:33.546613 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:33.546498 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:01:31 +0000 UTC" deadline="2028-04-18 03:20:14.764211196 +0000 UTC" Nov 03 22:06:33.546613 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:33.546591 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12749h13m41.217624397s" Nov 03 22:06:35.060214 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:35.060166 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:35.060975 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:35.060229 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:35.060975 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:35.060367 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:35.060975 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:35.060431 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs podName:393f28cb-5fc8-4b3e-8852-eebe41b42a2b nodeName:}" failed. No retries permitted until 2026-11-03 22:06:39.060412585 +0000 UTC m=+9.301681600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs") pod "network-metrics-daemon-sfqbq" (UID: "393f28cb-5fc8-4b3e-8852-eebe41b42a2b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:35.060975 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:35.060508 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:35.060975 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:35.060542 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret podName:99d8e137-a970-441d-9800-d1ce1c8f1f94 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:39.060531365 +0000 UTC m=+9.301800380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret") pod "global-pull-secret-syncer-8wmmt" (UID: "99d8e137-a970-441d-9800-d1ce1c8f1f94") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:35.161353 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:35.161269 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmppm\" (UniqueName: \"kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm\") pod \"network-check-target-z4ct8\" (UID: \"fab1061b-bf17-477e-bfdf-cb8b1d76b0ff\") " pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:35.161525 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:35.161463 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:06:35.161525 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:35.161479 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:06:35.161525 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:35.161488 2583 projected.go:196] Error preparing data for projected volume kube-api-access-jmppm for pod openshift-network-diagnostics/network-check-target-z4ct8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:35.161681 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:35.161532 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm podName:fab1061b-bf17-477e-bfdf-cb8b1d76b0ff nodeName:}" failed. No retries permitted until 2026-11-03 22:06:39.161519955 +0000 UTC m=+9.402788981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jmppm" (UniqueName: "kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm") pod "network-check-target-z4ct8" (UID: "fab1061b-bf17-477e-bfdf-cb8b1d76b0ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:35.277809 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:35.277752 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:35.278005 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:35.277873 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wmmt" podUID="99d8e137-a970-441d-9800-d1ce1c8f1f94" Nov 03 22:06:35.278005 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:35.277935 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:35.278005 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:35.277980 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4ct8" podUID="fab1061b-bf17-477e-bfdf-cb8b1d76b0ff" Nov 03 22:06:35.278198 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:35.278016 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:35.278198 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:35.278078 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfqbq" podUID="393f28cb-5fc8-4b3e-8852-eebe41b42a2b" Nov 03 22:06:37.278727 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:37.278116 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:37.278727 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:37.278473 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4ct8" podUID="fab1061b-bf17-477e-bfdf-cb8b1d76b0ff" Nov 03 22:06:37.278727 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:37.278352 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:37.278727 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:37.278564 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wmmt" podUID="99d8e137-a970-441d-9800-d1ce1c8f1f94" Nov 03 22:06:37.278727 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:37.278320 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:37.278727 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:37.278631 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfqbq" podUID="393f28cb-5fc8-4b3e-8852-eebe41b42a2b" Nov 03 22:06:37.335463 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:37.335406 2583 generic.go:356] "Generic (PLEG): container finished" podID="36effd34d282541d8cc7db737926b500" containerID="cd671c3474087a46cf1cd4397fe83dcb408ac6d0227be0d35263fb0cf893b4c6" exitCode=0 Nov 03 22:06:37.335653 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:37.335520 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" event={"ID":"36effd34d282541d8cc7db737926b500","Type":"ContainerDied","Data":"cd671c3474087a46cf1cd4397fe83dcb408ac6d0227be0d35263fb0cf893b4c6"} Nov 03 22:06:37.345678 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:37.345640 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal" event={"ID":"a3ff6dbfd235d6871a32270ec8f4d659","Type":"ContainerStarted","Data":"78578b8cf38372b5ee7fe3f069d91600f6d27b8436d0645ef341bcc6b9fb7c3c"} Nov 03 22:06:37.352379 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:37.352326 2583 generic.go:356] "Generic (PLEG): container finished" podID="b3e7a098-088b-4e25-ac35-dc531d619b79" containerID="032bbbf9cb5d077c1f5e6d4abb4bf65e1a53aebbd10dbd512baf7b62414963f9" exitCode=0 Nov 03 22:06:37.352379 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:37.352381 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2g9vv" event={"ID":"b3e7a098-088b-4e25-ac35-dc531d619b79","Type":"ContainerDied","Data":"032bbbf9cb5d077c1f5e6d4abb4bf65e1a53aebbd10dbd512baf7b62414963f9"} Nov 03 22:06:37.706451 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:37.706398 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal" podStartSLOduration=7.706383907 podStartE2EDuration="7.706383907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:37.588390282 +0000 UTC m=+7.829659318" watchObservedRunningTime="2026-11-03 22:06:37.706383907 +0000 UTC m=+7.947652941" Nov 03 22:06:38.361056 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:38.360368 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" event={"ID":"36effd34d282541d8cc7db737926b500","Type":"ContainerStarted","Data":"bcb3c40cf4180a2b6c7ba34cf7ba4b554c781a49afd687acdd0a60c89fc282ca"} Nov 03 22:06:38.364547 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:38.364458 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k4rnh" event={"ID":"19a326ff-4502-4590-9c0d-194f8cf14abd","Type":"ContainerStarted","Data":"26ea5aa934b47debefc681be680d4ee1c82536235c958135a985122f6b2d9631"} Nov 03 22:06:38.478571 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:38.478496 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" podStartSLOduration=8.478476057 podStartE2EDuration="8.478476057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:38.478286852 +0000 UTC m=+8.719555887" watchObservedRunningTime="2026-11-03 22:06:38.478476057 +0000 UTC m=+8.719745093" Nov 03 22:06:38.592491 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:38.592435 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k4rnh" podStartSLOduration=3.535517811 podStartE2EDuration="8.592421427s" totalImagesPullingTime="5.056903616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:38.591966604 +0000 UTC m=+8.833235643" watchObservedRunningTime="2026-11-03 22:06:38.592421427 +0000 UTC m=+8.833690464" Nov 03 22:06:39.106417 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:39.106345 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:39.106417 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:39.106392 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:39.106646 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:39.106515 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:39.106646 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:39.106577 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs podName:393f28cb-5fc8-4b3e-8852-eebe41b42a2b nodeName:}" failed. No retries permitted until 2026-11-03 22:06:47.106559119 +0000 UTC m=+17.347828138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs") pod "network-metrics-daemon-sfqbq" (UID: "393f28cb-5fc8-4b3e-8852-eebe41b42a2b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:39.106646 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:39.106544 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:39.106801 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:39.106656 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret podName:99d8e137-a970-441d-9800-d1ce1c8f1f94 nodeName:}" failed. No retries permitted until 2026-11-03 22:06:47.10662452 +0000 UTC m=+17.347893533 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret") pod "global-pull-secret-syncer-8wmmt" (UID: "99d8e137-a970-441d-9800-d1ce1c8f1f94") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:39.207404 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:39.207296 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmppm\" (UniqueName: \"kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm\") pod \"network-check-target-z4ct8\" (UID: \"fab1061b-bf17-477e-bfdf-cb8b1d76b0ff\") " pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:39.207587 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:39.207438 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:06:39.207587 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:39.207459 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:06:39.207587 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:39.207469 2583 projected.go:196] Error preparing data for projected volume kube-api-access-jmppm for pod openshift-network-diagnostics/network-check-target-z4ct8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:39.207587 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:39.207514 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm podName:fab1061b-bf17-477e-bfdf-cb8b1d76b0ff nodeName:}" failed. No retries permitted until 2026-11-03 22:06:47.207501582 +0000 UTC m=+17.448770596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jmppm" (UniqueName: "kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm") pod "network-check-target-z4ct8" (UID: "fab1061b-bf17-477e-bfdf-cb8b1d76b0ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:39.278573 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:39.278026 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:39.278573 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:39.278180 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wmmt" podUID="99d8e137-a970-441d-9800-d1ce1c8f1f94" Nov 03 22:06:39.278573 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:39.278263 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:39.278573 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:39.278337 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4ct8" podUID="fab1061b-bf17-477e-bfdf-cb8b1d76b0ff" Nov 03 22:06:39.278573 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:39.278401 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:39.278573 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:39.278494 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfqbq" podUID="393f28cb-5fc8-4b3e-8852-eebe41b42a2b" Nov 03 22:06:39.374785 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:39.373955 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t6b4b" event={"ID":"95246059-e5c1-4b42-9678-a1c8b2f63e80","Type":"ContainerStarted","Data":"07b93a43797bb7d44a1610135fb1cd5bfe8bb65c4fdfbed31259716561a94a39"} Nov 03 22:06:39.919367 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:39.919326 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k4rnh" Nov 03 22:06:39.920096 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:39.920071 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k4rnh" Nov 03 22:06:40.036663 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:40.036611 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t6b4b" podStartSLOduration=4.130512727 podStartE2EDuration="10.03659323s" totalImagesPullingTime="5.906080503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:39.487976384 +0000 UTC m=+9.729245492" watchObservedRunningTime="2026-11-03 22:06:40.03659323 +0000 UTC m=+10.277862267" Nov 03 22:06:40.379167 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:40.379089 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" event={"ID":"412af756-be10-401f-8413-419d66bafaad","Type":"ContainerStarted","Data":"87c67daa8dc0f22ac4fe32db97cafa152674eb4fd2dcf404e5d422768f20e365"} Nov 03 22:06:41.277919 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:41.277485 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:41.277919 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:41.277630 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wmmt" podUID="99d8e137-a970-441d-9800-d1ce1c8f1f94" Nov 03 22:06:41.277919 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:41.277706 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:41.277919 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:41.277785 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4ct8" podUID="fab1061b-bf17-477e-bfdf-cb8b1d76b0ff" Nov 03 22:06:41.277919 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:41.277830 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:41.277919 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:41.277878 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfqbq" podUID="393f28cb-5fc8-4b3e-8852-eebe41b42a2b" Nov 03 22:06:41.383933 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:41.383862 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:06:41.384865 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:41.384836 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jl87t" event={"ID":"37b1fdc9-35d8-4bd3-af0c-cea9299192e8","Type":"ContainerStarted","Data":"ffa263c91289c6ab08b4b1557a1c092778292936060b55de2fb3b6a638bf0d23"} Nov 03 22:06:41.763762 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:41.763705 2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 22:06:42.288607 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:42.288482 2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:06:41.763725716Z","UUID":"ef8375f0-944e-4b6d-9c70-25211d4d2ee1","Handler":null,"Name":"","Endpoint":""} Nov 03 22:06:42.290300 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:42.290269 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 22:06:42.290300 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:42.290305 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 22:06:42.387832 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:42.387795 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-79sq5" event={"ID":"866a1e7a-4728-4bd5-a88e-1a97f36f1eac","Type":"ContainerStarted","Data":"b5f1f4ab7602ac3e9fe19513eaa3c9516730e0c33dfd8baa9867ebae6b830f43"} Nov 03 22:06:42.390450 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:42.390385 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" event={"ID":"412af756-be10-401f-8413-419d66bafaad","Type":"ContainerStarted","Data":"f8d0d0153147041eb71eaa2de9b6da2a7c280e94cc16b59582d761a52bfaa558"} Nov 03 22:06:42.503949 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:42.503638 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-jl87t" podStartSLOduration=4.058138684 podStartE2EDuration="12.503614747s" totalImagesPullingTime="8.445476063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:41.501726955 +0000 UTC m=+11.742995995" watchObservedRunningTime="2026-11-03 22:06:42.503614747 +0000 UTC m=+12.744883852" Nov 03 22:06:43.278207 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:43.277652 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:43.278207 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:43.277820 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wmmt" podUID="99d8e137-a970-441d-9800-d1ce1c8f1f94" Nov 03 22:06:43.278207 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:43.277909 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:43.278207 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:43.277988 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4ct8" podUID="fab1061b-bf17-477e-bfdf-cb8b1d76b0ff" Nov 03 22:06:43.278207 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:43.278049 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:43.278207 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:43.278159 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfqbq" podUID="393f28cb-5fc8-4b3e-8852-eebe41b42a2b" Nov 03 22:06:43.395229 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:43.395187 2583 generic.go:356] "Generic (PLEG): container finished" podID="b3e7a098-088b-4e25-ac35-dc531d619b79" containerID="285673180f92493bf67be500a883a8e342236e7c34106cb14543435cf31285c9" exitCode=0 Nov 03 22:06:43.395700 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:43.395271 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2g9vv" event={"ID":"b3e7a098-088b-4e25-ac35-dc531d619b79","Type":"ContainerDied","Data":"285673180f92493bf67be500a883a8e342236e7c34106cb14543435cf31285c9"} Nov 03 22:06:43.397190 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:43.397164 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gsmpr" event={"ID":"f4d66c00-b01b-4c14-998a-d84707b4600b","Type":"ContainerStarted","Data":"2cfeb180e472881f23237eb4c7e87dcdd2c1ed5dbaca5ebe8e48ebd522dc6ec1"} Nov 03 22:06:43.617208 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:43.617008 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-79sq5" podStartSLOduration=4.7427032449999995 podStartE2EDuration="13.61699293s" totalImagesPullingTime="8.874289685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:42.502942256 +0000 UTC m=+12.744211290" watchObservedRunningTime="2026-11-03 22:06:43.61699293 +0000 UTC m=+13.858261964" Nov 03 22:06:44.401900 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:44.401853 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" event={"ID":"412af756-be10-401f-8413-419d66bafaad","Type":"ContainerStarted","Data":"7a9ff81203dad485719f98e3a1607f14101355028f4b6697f4030e8ef63c9573"} Nov 03 22:06:44.403574 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:44.403542 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" event={"ID":"8b599076-85db-4b98-8636-485e964a5e59","Type":"ContainerStarted","Data":"64c8764e28004da5f426ec29d749c0f61f16794d465bedfa0fffac57fb3907bd"} Nov 03 22:06:44.518726 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:44.518667 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gsmpr" podStartSLOduration=5.443334724 podStartE2EDuration="14.518650898s" totalImagesPullingTime="9.075316174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:43.678417216 +0000 UTC m=+13.919686253" watchObservedRunningTime="2026-11-03 22:06:44.518650898 +0000 UTC m=+14.759919933" Nov 03 22:06:44.518892 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:44.518800 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" podStartSLOduration=4.097738186 podStartE2EDuration="14.518791286s" totalImagesPullingTime="10.4210531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:44.518567856 +0000 UTC m=+14.759836891" watchObservedRunningTime="2026-11-03 22:06:44.518791286 +0000 UTC m=+14.760060322" Nov 03 22:06:44.635530 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:44.635471 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" podStartSLOduration=3.751435099 podStartE2EDuration="14.635455104s" totalImagesPullingTime="10.884020005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:44.635176213 +0000 UTC m=+14.876445247" watchObservedRunningTime="2026-11-03 22:06:44.635455104 +0000 UTC m=+14.876724139" Nov 03 22:06:45.278199 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:45.278161 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:45.278388 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:45.278173 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:45.278388 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:45.278275 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4ct8" podUID="fab1061b-bf17-477e-bfdf-cb8b1d76b0ff" Nov 03 22:06:45.278388 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:45.278180 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:45.278561 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:45.278386 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfqbq" podUID="393f28cb-5fc8-4b3e-8852-eebe41b42a2b" Nov 03 22:06:45.278561 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:45.278439 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wmmt" podUID="99d8e137-a970-441d-9800-d1ce1c8f1f94" Nov 03 22:06:45.408537 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:45.408495 2583 generic.go:356] "Generic (PLEG): container finished" podID="b3e7a098-088b-4e25-ac35-dc531d619b79" containerID="ece83f952cd0ebe944fa341f644eb179f62116b317eda48acf0794b07ce367c7" exitCode=0 Nov 03 22:06:45.408537 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:45.408521 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2g9vv" event={"ID":"b3e7a098-088b-4e25-ac35-dc531d619b79","Type":"ContainerDied","Data":"ece83f952cd0ebe944fa341f644eb179f62116b317eda48acf0794b07ce367c7"} Nov 03 22:06:47.177859 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:47.177773 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:47.177859 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:47.177823 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:47.178560 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:47.177948 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:47.178560 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:47.178036 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret podName:99d8e137-a970-441d-9800-d1ce1c8f1f94 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:03.178013213 +0000 UTC m=+33.419282242 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret") pod "global-pull-secret-syncer-8wmmt" (UID: "99d8e137-a970-441d-9800-d1ce1c8f1f94") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:06:47.178560 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:47.177953 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:47.178560 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:47.178133 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs podName:393f28cb-5fc8-4b3e-8852-eebe41b42a2b nodeName:}" failed. No retries permitted until 2026-11-03 22:07:03.178115271 +0000 UTC m=+33.419384293 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs") pod "network-metrics-daemon-sfqbq" (UID: "393f28cb-5fc8-4b3e-8852-eebe41b42a2b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:06:47.277401 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:47.277361 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:47.277580 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:47.277464 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:47.277580 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:47.277499 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:47.277660 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:47.277641 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wmmt" podUID="99d8e137-a970-441d-9800-d1ce1c8f1f94" Nov 03 22:06:47.277770 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:47.277750 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfqbq" podUID="393f28cb-5fc8-4b3e-8852-eebe41b42a2b" Nov 03 22:06:47.277840 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:47.277823 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4ct8" podUID="fab1061b-bf17-477e-bfdf-cb8b1d76b0ff" Nov 03 22:06:47.278486 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:47.278465 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmppm\" (UniqueName: \"kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm\") pod \"network-check-target-z4ct8\" (UID: \"fab1061b-bf17-477e-bfdf-cb8b1d76b0ff\") " pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:47.278613 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:47.278598 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:06:47.278668 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:47.278621 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:06:47.278668 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:47.278633 2583 projected.go:196] Error preparing data for projected volume kube-api-access-jmppm for pod openshift-network-diagnostics/network-check-target-z4ct8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:47.278750 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:47.278690 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm podName:fab1061b-bf17-477e-bfdf-cb8b1d76b0ff nodeName:}" failed. No retries permitted until 2026-11-03 22:07:03.278673198 +0000 UTC m=+33.519942227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jmppm" (UniqueName: "kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm") pod "network-check-target-z4ct8" (UID: "fab1061b-bf17-477e-bfdf-cb8b1d76b0ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:06:47.416241 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:47.416207 2583 generic.go:356] "Generic (PLEG): container finished" podID="b3e7a098-088b-4e25-ac35-dc531d619b79" containerID="85cb19323313b002fced32c2972cf2b67a0eea1a0147d113b761fcb9e0bf33a3" exitCode=0 Nov 03 22:06:47.416411 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:47.416276 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2g9vv" event={"ID":"b3e7a098-088b-4e25-ac35-dc531d619b79","Type":"ContainerDied","Data":"85cb19323313b002fced32c2972cf2b67a0eea1a0147d113b761fcb9e0bf33a3"} Nov 03 22:06:48.421798 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:48.421436 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" event={"ID":"1d49e0d2-566e-472c-9c2d-c85256d52de4","Type":"ContainerStarted","Data":"2de674ad7c944cdc852c81b6e4f7c549f926c2f0edf85d110035b78bc2503b65"} Nov 03 22:06:48.422183 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:48.421797 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" event={"ID":"1d49e0d2-566e-472c-9c2d-c85256d52de4","Type":"ContainerStarted","Data":"2815d20ade4dabf109023529c95608b64b7e24cd878066f9834d9b8563daa3a6"} Nov 03 22:06:48.422183 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:48.421831 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" event={"ID":"1d49e0d2-566e-472c-9c2d-c85256d52de4","Type":"ContainerStarted","Data":"70960deacc094fe536622a1a6befb4578818cb16ac4e79b842204888b0fd0e68"} Nov 03 22:06:48.422183 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:48.421857 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" event={"ID":"1d49e0d2-566e-472c-9c2d-c85256d52de4","Type":"ContainerStarted","Data":"a80c6adf7e04355924777f98af1365d554f71c71b0509d9f41dde9851721fb24"} Nov 03 22:06:48.422183 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:48.421883 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" event={"ID":"1d49e0d2-566e-472c-9c2d-c85256d52de4","Type":"ContainerStarted","Data":"47ced082b0eb9eeacb9d3c58e97155e64567b20ad3ed54059435036ff3a6fecc"} Nov 03 22:06:48.422183 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:48.421909 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" event={"ID":"1d49e0d2-566e-472c-9c2d-c85256d52de4","Type":"ContainerStarted","Data":"447c714e62ad707f6288a607b346f23cee0661a933e7ace0c3c19b58d013d613"} Nov 03 22:06:49.277610 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:49.277567 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:49.277610 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:49.277601 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:49.277884 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:49.277745 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4ct8" podUID="fab1061b-bf17-477e-bfdf-cb8b1d76b0ff" Nov 03 22:06:49.277884 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:49.277845 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:49.277884 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:49.277858 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfqbq" podUID="393f28cb-5fc8-4b3e-8852-eebe41b42a2b" Nov 03 22:06:49.277997 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:49.277917 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wmmt" podUID="99d8e137-a970-441d-9800-d1ce1c8f1f94" Nov 03 22:06:50.429682 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:50.429606 2583 generic.go:356] "Generic (PLEG): container finished" podID="b3e7a098-088b-4e25-ac35-dc531d619b79" containerID="32163c437a1fc40243027f943f77c401a1e5d27b71e53b214601855c272f7230" exitCode=0 Nov 03 22:06:50.430385 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:50.429694 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2g9vv" event={"ID":"b3e7a098-088b-4e25-ac35-dc531d619b79","Type":"ContainerDied","Data":"32163c437a1fc40243027f943f77c401a1e5d27b71e53b214601855c272f7230"} Nov 03 22:06:50.433184 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:50.433157 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" event={"ID":"1d49e0d2-566e-472c-9c2d-c85256d52de4","Type":"ContainerStarted","Data":"be7371e803138a15314c0a596112e170011072368f30d4367ecdafa803c3df59"} Nov 03 22:06:51.278320 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:51.278288 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:51.278500 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:51.278297 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:51.278500 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:51.278393 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wmmt" podUID="99d8e137-a970-441d-9800-d1ce1c8f1f94" Nov 03 22:06:51.278500 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:51.278297 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:51.278500 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:51.278465 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4ct8" podUID="fab1061b-bf17-477e-bfdf-cb8b1d76b0ff" Nov 03 22:06:51.278643 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:51.278526 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfqbq" podUID="393f28cb-5fc8-4b3e-8852-eebe41b42a2b" Nov 03 22:06:51.442003 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:51.441966 2583 generic.go:356] "Generic (PLEG): container finished" podID="b3e7a098-088b-4e25-ac35-dc531d619b79" containerID="90780cb57bb2d16c60232303dedb99aa7f663e863d75e85d1a364f686926080f" exitCode=0 Nov 03 22:06:51.442412 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:51.442013 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2g9vv" event={"ID":"b3e7a098-088b-4e25-ac35-dc531d619b79","Type":"ContainerDied","Data":"90780cb57bb2d16c60232303dedb99aa7f663e863d75e85d1a364f686926080f"} Nov 03 22:06:52.448878 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:52.448621 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" event={"ID":"1d49e0d2-566e-472c-9c2d-c85256d52de4","Type":"ContainerStarted","Data":"17ef432854ccae88508b51eeb594ee310f59c3074f7deb58fb5c1837bea32518"} Nov 03 22:06:52.449607 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:52.448939 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:52.449607 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:52.448993 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:52.449607 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:52.449040 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:52.452903 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:52.452851 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2g9vv" event={"ID":"b3e7a098-088b-4e25-ac35-dc531d619b79","Type":"ContainerStarted","Data":"4d6f989a54ad27b3b70a6f70a37a29fb615ba3ac8b8da6a1755c017da6cd069d"} Nov 03 22:06:52.466965 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:52.466854 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:52.467138 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:52.467119 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:06:52.707974 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:52.707914 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" podStartSLOduration=7.808088238 podStartE2EDuration="22.707900194s" totalImagesPullingTime="14.899811956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:52.671751904 +0000 UTC m=+22.913020941" watchObservedRunningTime="2026-11-03 22:06:52.707900194 +0000 UTC m=+22.949169229" Nov 03 22:06:52.790720 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:52.790612 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2g9vv" podStartSLOduration=9.173228494 podStartE2EDuration="22.79060166s" totalImagesPullingTime="13.617373166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:06:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:06:52.790482452 +0000 UTC m=+23.031751488" watchObservedRunningTime="2026-11-03 22:06:52.79060166 +0000 UTC m=+23.031870695" Nov 03 22:06:52.821821 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:52.821790 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k4rnh" Nov 03 22:06:52.822016 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:52.821927 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:06:52.822460 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:52.822436 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k4rnh" Nov 03 22:06:53.277406 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:53.277324 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:53.277552 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:53.277447 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:53.277552 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:53.277453 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wmmt" podUID="99d8e137-a970-441d-9800-d1ce1c8f1f94" Nov 03 22:06:53.277552 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:53.277479 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:53.277644 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:53.277561 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfqbq" podUID="393f28cb-5fc8-4b3e-8852-eebe41b42a2b" Nov 03 22:06:53.277644 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:53.277629 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4ct8" podUID="fab1061b-bf17-477e-bfdf-cb8b1d76b0ff" Nov 03 22:06:55.003214 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:55.003173 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8wmmt"] Nov 03 22:06:55.003796 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:55.003298 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:55.003796 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:55.003410 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wmmt" podUID="99d8e137-a970-441d-9800-d1ce1c8f1f94" Nov 03 22:06:55.006109 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:55.006058 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sfqbq"] Nov 03 22:06:55.006231 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:55.006204 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:55.006340 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:55.006320 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfqbq" podUID="393f28cb-5fc8-4b3e-8852-eebe41b42a2b" Nov 03 22:06:55.054297 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:55.054269 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z4ct8"] Nov 03 22:06:55.054403 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:55.054368 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:55.054459 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:55.054441 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4ct8" podUID="fab1061b-bf17-477e-bfdf-cb8b1d76b0ff" Nov 03 22:06:57.278187 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:57.278148 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:57.278757 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:57.278150 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:57.278757 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:57.278279 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wmmt" podUID="99d8e137-a970-441d-9800-d1ce1c8f1f94" Nov 03 22:06:57.278757 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:57.278148 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:57.278757 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:57.278336 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfqbq" podUID="393f28cb-5fc8-4b3e-8852-eebe41b42a2b" Nov 03 22:06:57.278757 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:57.278402 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4ct8" podUID="fab1061b-bf17-477e-bfdf-cb8b1d76b0ff" Nov 03 22:06:59.278046 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:59.278009 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:06:59.278046 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:59.278039 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:06:59.278505 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:06:59.278036 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:06:59.278505 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:59.278133 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8wmmt" podUID="99d8e137-a970-441d-9800-d1ce1c8f1f94" Nov 03 22:06:59.278505 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:59.278228 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sfqbq" podUID="393f28cb-5fc8-4b3e-8852-eebe41b42a2b" Nov 03 22:06:59.278505 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:06:59.278312 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z4ct8" podUID="fab1061b-bf17-477e-bfdf-cb8b1d76b0ff" Nov 03 22:07:01.055423 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.055237 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeReady" Nov 03 22:07:01.055918 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.055522 2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 22:07:01.213822 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.213788 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f88sj"] Nov 03 22:07:01.216795 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.216769 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tc9n2"] Nov 03 22:07:01.216970 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.216924 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-f88sj" Nov 03 22:07:01.219403 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.219376 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-z2q8s"] Nov 03 22:07:01.219595 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.219577 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tc9n2" Nov 03 22:07:01.221853 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.221831 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-p6mft"] Nov 03 22:07:01.221975 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.221955 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-z2q8s" Nov 03 22:07:01.224233 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.224212 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6nsjz"] Nov 03 22:07:01.224351 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.224339 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:01.226685 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.226661 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fqp5x"] Nov 03 22:07:01.227144 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.226864 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6nsjz" Nov 03 22:07:01.230502 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.230480 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb"] Nov 03 22:07:01.230819 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.230802 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.233888 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.233864 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf"] Nov 03 22:07:01.234019 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.233999 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" Nov 03 22:07:01.236584 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.236564 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-9mgrk"] Nov 03 22:07:01.236720 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.236699 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:01.239626 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.239574 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx"] Nov 03 22:07:01.245510 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.239711 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:01.245510 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.243440 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-595c4958c9-2rswm"] Nov 03 22:07:01.245510 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.243823 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:01.250165 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.250136 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.265857 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.265838 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z"] Nov 03 22:07:01.268614 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.268588 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-84d846b7f5-4qjqn"] Nov 03 22:07:01.268751 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.268719 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" Nov 03 22:07:01.271797 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.271762 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-q4b9r"] Nov 03 22:07:01.271934 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.271913 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:01.276480 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.276449 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58d66d58db-phhsh"] Nov 03 22:07:01.282291 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.282263 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q4b9r" Nov 03 22:07:01.284178 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.284150 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-w55xl"] Nov 03 22:07:01.284325 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.284224 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfhnj\" (UniqueName: \"kubernetes.io/projected/fe0af137-f9ce-4f18-8329-3f7df6a728bb-kube-api-access-bfhnj\") pod \"kas-connection-checker-685895c7c9-f88sj\" (UID: \"fe0af137-f9ce-4f18-8329-3f7df6a728bb\") " pod="kube-system/kas-connection-checker-685895c7c9-f88sj" Nov 03 22:07:01.284500 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.284475 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.284614 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.284505 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:07:01.284614 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.284512 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:07:01.284998 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.284981 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:07:01.289044 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.288634 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rntwm"] Nov 03 22:07:01.289467 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.289446 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w55xl" Nov 03 22:07:01.291986 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.291965 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f88sj"] Nov 03 22:07:01.292186 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.292172 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tc9n2"] Nov 03 22:07:01.292341 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.292327 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-p6mft"] Nov 03 22:07:01.293164 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.292090 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:01.293302 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.293051 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-z2q8s"] Nov 03 22:07:01.293421 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.293409 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb"] Nov 03 22:07:01.293576 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.293554 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fqp5x"] Nov 03 22:07:01.293660 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.293582 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx"] Nov 03 22:07:01.293660 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.293602 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf"] Nov 03 22:07:01.293660 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.293616 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z"] Nov 03 22:07:01.293660 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.293629 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-595c4958c9-2rswm"] Nov 03 22:07:01.293660 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.293642 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rntwm"] Nov 03 22:07:01.317090 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.315518 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 22:07:01.317090 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.315563 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 22:07:01.317090 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.315566 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 22:07:01.317090 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.316833 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-84d846b7f5-4qjqn"] Nov 03 22:07:01.317708 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.317676 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58d66d58db-phhsh"] Nov 03 22:07:01.318340 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.318324 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6nsjz"] Nov 03 22:07:01.318895 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.318871 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w55xl"] Nov 03 22:07:01.319514 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.319499 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-9mgrk"] Nov 03 22:07:01.320112 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.320094 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-q4b9r"] Nov 03 22:07:01.371005 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.370975 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 22:07:01.371183 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.371113 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 22:07:01.371183 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.371171 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 22:07:01.371183 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.371180 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 22:07:01.371325 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.371180 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 22:07:01.371325 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.371182 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 22:07:01.371325 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.371218 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 22:07:01.371325 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.371237 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 22:07:01.371503 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.371380 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 22:07:01.371503 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.371481 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:07:01.371833 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.371811 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 22:07:01.371931 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.371822 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:07:01.371931 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.371824 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:07:01.371931 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.371829 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 22:07:01.371931 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.371851 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:07:01.372193 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.371874 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 22:07:01.372193 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.372088 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 22:07:01.372193 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.372110 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 22:07:01.372193 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.372135 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:07:01.372193 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.372111 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:07:01.372193 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.372169 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 22:07:01.372425 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.372198 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 22:07:01.372425 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.372204 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 22:07:01.372425 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.372261 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:07:01.372425 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.372273 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 22:07:01.372425 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.372330 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 22:07:01.372628 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.372434 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 22:07:01.372859 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.372841 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 22:07:01.384856 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.384815 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-image-registry-private-configuration\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.384856 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.384860 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-default-certificate\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:01.385038 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.384888 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d1498363-2c98-4046-8feb-abb772a80149-config\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:01.385038 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.384969 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptsfs\" (UniqueName: \"kubernetes.io/projected/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-kube-api-access-ptsfs\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:01.385038 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385017 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdh26\" (UniqueName: \"kubernetes.io/projected/ec658817-ab95-4b41-a457-d067b9754e3e-kube-api-access-wdh26\") pod \"kas-connection-checker-685895c7c9-tc9n2\" (UID: \"ec658817-ab95-4b41-a457-d067b9754e3e\") " pod="kube-system/kas-connection-checker-685895c7c9-tc9n2" Nov 03 22:07:01.385038 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385037 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgntw\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-kube-api-access-qgntw\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.385262 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385116 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b5a34d-6354-4586-9dd1-55279e0dcfad-tmp-dir\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:01.385262 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385148 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z\" (UID: \"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" Nov 03 22:07:01.385262 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385178 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v728z\" (UniqueName: \"kubernetes.io/projected/43929f2c-6b7a-4f25-8a97-51f6bee5481a-kube-api-access-v728z\") pod \"volume-data-source-validator-58c9f5f46d-6nsjz\" (UID: \"43929f2c-6b7a-4f25-8a97-51f6bee5481a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6nsjz" Nov 03 22:07:01.385262 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385224 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:01.385262 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385252 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-installation-pull-secrets\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.385460 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385277 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c0ac5b4-308e-4eca-8301-2c470cecdbef-ca-trust-extracted\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.385460 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385303 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3b08229-d363-4336-bb37-89df96b43dbe-cert\") pod \"ingress-canary-w55xl\" (UID: \"d3b08229-d363-4336-bb37-89df96b43dbe\") " pod="openshift-ingress-canary/ingress-canary-w55xl" Nov 03 22:07:01.385460 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385347 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/60e01c32-cde7-4c5e-add4-ca8c17e82e34-snapshots\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.385460 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385380 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-installation-pull-secrets\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.385460 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385400 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:01.385665 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385458 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1498363-2c98-4046-8feb-abb772a80149-trusted-ca\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:01.385665 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385490 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-trusted-ca\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.385665 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385519 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e099b6bc-addb-428b-9a88-2c8b8e544f46-serving-cert\") pod \"service-ca-operator-544d749c8c-96jwb\" (UID: \"e099b6bc-addb-428b-9a88-2c8b8e544f46\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" Nov 03 22:07:01.385665 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385568 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1498363-2c98-4046-8feb-abb772a80149-serving-cert\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:01.385665 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385588 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms2cm\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-kube-api-access-ms2cm\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.385665 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385620 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8cv4\" (UniqueName: \"kubernetes.io/projected/43b5a34d-6354-4586-9dd1-55279e0dcfad-kube-api-access-m8cv4\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:01.385665 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385659 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:01.385943 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385692 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-bound-sa-token\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.385943 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385751 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.385943 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385776 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:01.385943 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385795 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:01.385943 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385825 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e01c32-cde7-4c5e-add4-ca8c17e82e34-service-ca-bundle\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.385943 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385862 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-stats-auth\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:01.385943 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385916 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43b5a34d-6354-4586-9dd1-55279e0dcfad-config-volume\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:01.386279 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385954 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p6mft\" (UID: \"0a4cedce-72c5-4357-8001-23a5e14d7d9d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:01.386279 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.385990 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx9rl\" (UniqueName: \"kubernetes.io/projected/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3-kube-api-access-fx9rl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z\" (UID: \"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" Nov 03 22:07:01.386279 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386047 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf54n\" (UniqueName: \"kubernetes.io/projected/d3b08229-d363-4336-bb37-89df96b43dbe-kube-api-access-gf54n\") pod \"ingress-canary-w55xl\" (UID: \"d3b08229-d363-4336-bb37-89df96b43dbe\") " pod="openshift-ingress-canary/ingress-canary-w55xl" Nov 03 22:07:01.386279 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386105 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzd9z\" (UniqueName: \"kubernetes.io/projected/508293cb-d657-4cfb-a67a-0155a44d6e47-kube-api-access-nzd9z\") pod \"kas-connection-checker-685895c7c9-z2q8s\" (UID: \"508293cb-d657-4cfb-a67a-0155a44d6e47\") " pod="kube-system/kas-connection-checker-685895c7c9-z2q8s" Nov 03 22:07:01.386279 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386138 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-certificates\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.386279 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386165 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e01c32-cde7-4c5e-add4-ca8c17e82e34-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.386279 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386226 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfhnj\" (UniqueName: \"kubernetes.io/projected/fe0af137-f9ce-4f18-8329-3f7df6a728bb-kube-api-access-bfhnj\") pod \"kas-connection-checker-685895c7c9-f88sj\" (UID: \"fe0af137-f9ce-4f18-8329-3f7df6a728bb\") " pod="kube-system/kas-connection-checker-685895c7c9-f88sj" Nov 03 22:07:01.386279 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386260 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjj86\" (UniqueName: \"kubernetes.io/projected/d1498363-2c98-4046-8feb-abb772a80149-kube-api-access-tjj86\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:01.386279 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386279 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.386636 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386305 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aacf527d-b30b-48cd-b5fd-be3eba9a2095-ca-trust-extracted\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.386636 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386349 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-bound-sa-token\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.386636 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386367 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e099b6bc-addb-428b-9a88-2c8b8e544f46-config\") pod \"service-ca-operator-544d749c8c-96jwb\" (UID: \"e099b6bc-addb-428b-9a88-2c8b8e544f46\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" Nov 03 22:07:01.386636 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386384 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0a4cedce-72c5-4357-8001-23a5e14d7d9d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-p6mft\" (UID: \"0a4cedce-72c5-4357-8001-23a5e14d7d9d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:01.386636 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386417 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-trusted-ca\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.386636 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386438 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9kvp\" (UniqueName: \"kubernetes.io/projected/e099b6bc-addb-428b-9a88-2c8b8e544f46-kube-api-access-b9kvp\") pod \"service-ca-operator-544d749c8c-96jwb\" (UID: \"e099b6bc-addb-428b-9a88-2c8b8e544f46\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" Nov 03 22:07:01.386636 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386464 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:01.386636 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386500 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e01c32-cde7-4c5e-add4-ca8c17e82e34-tmp\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.386636 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386523 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z\" (UID: \"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" Nov 03 22:07:01.386636 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386562 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grqtw\" (UniqueName: \"kubernetes.io/projected/2e0d6adb-5887-42af-ad83-48863e1ac386-kube-api-access-grqtw\") pod \"network-check-source-6f78d59f99-q4b9r\" (UID: \"2e0d6adb-5887-42af-ad83-48863e1ac386\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q4b9r" Nov 03 22:07:01.386636 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386588 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ab2929c7-ab47-4660-80eb-f973bf6efdae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:01.386636 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386623 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whwxk\" (UniqueName: \"kubernetes.io/projected/ab2929c7-ab47-4660-80eb-f973bf6efdae-kube-api-access-whwxk\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:01.386985 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386694 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60e01c32-cde7-4c5e-add4-ca8c17e82e34-serving-cert\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.386985 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386711 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz8gb\" (UniqueName: \"kubernetes.io/projected/60e01c32-cde7-4c5e-add4-ca8c17e82e34-kube-api-access-kz8gb\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.386985 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386727 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zt5p\" (UniqueName: \"kubernetes.io/projected/f2a09669-a5d7-4dbb-9ddf-842f4f919028-kube-api-access-8zt5p\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:01.386985 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386743 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-certificates\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.386985 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.386758 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-image-registry-private-configuration\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.426989 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.426952 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 22:07:01.428289 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.427715 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 22:07:01.428289 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.427778 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 22:07:01.428289 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.427838 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 22:07:01.428289 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.427910 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 22:07:01.428289 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.427910 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 22:07:01.428289 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.427948 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 22:07:01.428289 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.427993 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 22:07:01.428289 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.428049 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 22:07:01.428289 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.428131 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:07:01.428289 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.428130 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kxpq\"" Nov 03 22:07:01.428289 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.428148 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 22:07:01.428289 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.428182 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 22:07:01.428289 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.428186 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 22:07:01.428289 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.428132 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 22:07:01.428289 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.428219 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-g6ff6\"" Nov 03 22:07:01.428289 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.428257 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:07:01.428289 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.428258 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 22:07:01.428289 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.428273 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2d6mp\"" Nov 03 22:07:01.428895 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.428318 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fcb7n\"" Nov 03 22:07:01.428895 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.428341 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 22:07:01.428895 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.428389 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ng2g2\"" Nov 03 22:07:01.428895 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.428404 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 22:07:01.428895 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.428427 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gsv4s\"" Nov 03 22:07:01.428895 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.428466 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 22:07:01.428895 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.428468 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 22:07:01.446249 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.446213 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfhnj\" (UniqueName: \"kubernetes.io/projected/fe0af137-f9ce-4f18-8329-3f7df6a728bb-kube-api-access-bfhnj\") pod \"kas-connection-checker-685895c7c9-f88sj\" (UID: \"fe0af137-f9ce-4f18-8329-3f7df6a728bb\") " pod="kube-system/kas-connection-checker-685895c7c9-f88sj" Nov 03 22:07:01.487350 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487312 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60e01c32-cde7-4c5e-add4-ca8c17e82e34-serving-cert\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.487350 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487347 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz8gb\" (UniqueName: \"kubernetes.io/projected/60e01c32-cde7-4c5e-add4-ca8c17e82e34-kube-api-access-kz8gb\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.487624 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487367 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zt5p\" (UniqueName: \"kubernetes.io/projected/f2a09669-a5d7-4dbb-9ddf-842f4f919028-kube-api-access-8zt5p\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:01.487624 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487383 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-certificates\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.487624 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487515 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-image-registry-private-configuration\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.487624 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487551 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-image-registry-private-configuration\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.487624 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487574 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-default-certificate\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:01.487624 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487603 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d1498363-2c98-4046-8feb-abb772a80149-config\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:01.487624 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487625 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptsfs\" (UniqueName: \"kubernetes.io/projected/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-kube-api-access-ptsfs\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:01.487962 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487651 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdh26\" (UniqueName: \"kubernetes.io/projected/ec658817-ab95-4b41-a457-d067b9754e3e-kube-api-access-wdh26\") pod \"kas-connection-checker-685895c7c9-tc9n2\" (UID: \"ec658817-ab95-4b41-a457-d067b9754e3e\") " pod="kube-system/kas-connection-checker-685895c7c9-tc9n2" Nov 03 22:07:01.487962 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487673 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgntw\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-kube-api-access-qgntw\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.487962 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487696 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b5a34d-6354-4586-9dd1-55279e0dcfad-tmp-dir\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:01.487962 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487722 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z\" (UID: \"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" Nov 03 22:07:01.487962 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487745 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v728z\" (UniqueName: \"kubernetes.io/projected/43929f2c-6b7a-4f25-8a97-51f6bee5481a-kube-api-access-v728z\") pod \"volume-data-source-validator-58c9f5f46d-6nsjz\" (UID: \"43929f2c-6b7a-4f25-8a97-51f6bee5481a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6nsjz" Nov 03 22:07:01.487962 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487773 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:01.487962 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487800 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-installation-pull-secrets\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.487962 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487826 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c0ac5b4-308e-4eca-8301-2c470cecdbef-ca-trust-extracted\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.487962 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487849 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3b08229-d363-4336-bb37-89df96b43dbe-cert\") pod \"ingress-canary-w55xl\" (UID: \"d3b08229-d363-4336-bb37-89df96b43dbe\") " pod="openshift-ingress-canary/ingress-canary-w55xl" Nov 03 22:07:01.487962 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487871 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/60e01c32-cde7-4c5e-add4-ca8c17e82e34-snapshots\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.487962 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487899 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-installation-pull-secrets\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.487962 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487927 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:01.487962 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487957 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1498363-2c98-4046-8feb-abb772a80149-trusted-ca\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:01.488580 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.487981 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-trusted-ca\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.488580 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.488010 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e099b6bc-addb-428b-9a88-2c8b8e544f46-serving-cert\") pod \"service-ca-operator-544d749c8c-96jwb\" (UID: \"e099b6bc-addb-428b-9a88-2c8b8e544f46\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" Nov 03 22:07:01.488580 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.488046 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1498363-2c98-4046-8feb-abb772a80149-serving-cert\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:01.488580 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.488097 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms2cm\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-kube-api-access-ms2cm\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.488580 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.488126 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8cv4\" (UniqueName: \"kubernetes.io/projected/43b5a34d-6354-4586-9dd1-55279e0dcfad-kube-api-access-m8cv4\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:01.488580 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.488171 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:01.488580 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.488186 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-certificates\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.488580 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.488199 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-bound-sa-token\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.488974 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.488778 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:07:01.489153 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.489127 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d1498363-2c98-4046-8feb-abb772a80149-config\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:01.491349 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.489269 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b5a34d-6354-4586-9dd1-55279e0dcfad-tmp-dir\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:01.491349 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.489480 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/60e01c32-cde7-4c5e-add4-ca8c17e82e34-snapshots\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.491349 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.488782 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle podName:b537a9eb-5abf-4f6b-9a35-0f13c5211bf2 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:01.988763921 +0000 UTC m=+32.230032958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle") pod "router-default-84d846b7f5-4qjqn" (UID: "b537a9eb-5abf-4f6b-9a35-0f13c5211bf2") : configmap references non-existent config key: service-ca.crt Nov 03 22:07:01.491349 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.489541 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs podName:b537a9eb-5abf-4f6b-9a35-0f13c5211bf2 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:01.989522187 +0000 UTC m=+32.230791214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs") pod "router-default-84d846b7f5-4qjqn" (UID: "b537a9eb-5abf-4f6b-9a35-0f13c5211bf2") : secret "router-metrics-certs-default" not found Nov 03 22:07:01.491349 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.489590 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.491349 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.489623 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:01.491349 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.489653 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:01.491349 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.489697 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e01c32-cde7-4c5e-add4-ca8c17e82e34-service-ca-bundle\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.491349 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.489733 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-stats-auth\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:01.491349 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.489761 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43b5a34d-6354-4586-9dd1-55279e0dcfad-config-volume\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:01.491349 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.489781 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z\" (UID: \"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" Nov 03 22:07:01.491349 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.489790 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p6mft\" (UID: \"0a4cedce-72c5-4357-8001-23a5e14d7d9d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:01.491349 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.489822 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx9rl\" (UniqueName: \"kubernetes.io/projected/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3-kube-api-access-fx9rl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z\" (UID: \"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" Nov 03 22:07:01.491349 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.489854 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf54n\" (UniqueName: \"kubernetes.io/projected/d3b08229-d363-4336-bb37-89df96b43dbe-kube-api-access-gf54n\") pod \"ingress-canary-w55xl\" (UID: \"d3b08229-d363-4336-bb37-89df96b43dbe\") " pod="openshift-ingress-canary/ingress-canary-w55xl" Nov 03 22:07:01.491349 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.489878 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:01.492206 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.489888 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzd9z\" (UniqueName: \"kubernetes.io/projected/508293cb-d657-4cfb-a67a-0155a44d6e47-kube-api-access-nzd9z\") pod \"kas-connection-checker-685895c7c9-z2q8s\" (UID: \"508293cb-d657-4cfb-a67a-0155a44d6e47\") " pod="kube-system/kas-connection-checker-685895c7c9-z2q8s" Nov 03 22:07:01.492206 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.489917 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-certificates\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.492206 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.489932 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls podName:ab2929c7-ab47-4660-80eb-f973bf6efdae nodeName:}" failed. No retries permitted until 2026-11-03 22:07:01.989916034 +0000 UTC m=+32.231185061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8gtxf" (UID: "ab2929c7-ab47-4660-80eb-f973bf6efdae") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:01.492206 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.489966 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e01c32-cde7-4c5e-add4-ca8c17e82e34-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.492206 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.490017 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjj86\" (UniqueName: \"kubernetes.io/projected/d1498363-2c98-4046-8feb-abb772a80149-kube-api-access-tjj86\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:01.492206 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.490045 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.492206 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.490045 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1498363-2c98-4046-8feb-abb772a80149-trusted-ca\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:01.492206 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.490154 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:07:01.492206 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.490173 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595c4958c9-2rswm: secret "image-registry-tls" not found Nov 03 22:07:01.492206 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.490227 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls podName:7c0ac5b4-308e-4eca-8301-2c470cecdbef nodeName:}" failed. No retries permitted until 2026-11-03 22:07:01.99021197 +0000 UTC m=+32.231481003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls") pod "image-registry-595c4958c9-2rswm" (UID: "7c0ac5b4-308e-4eca-8301-2c470cecdbef") : secret "image-registry-tls" not found Nov 03 22:07:01.492206 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.490254 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aacf527d-b30b-48cd-b5fd-be3eba9a2095-ca-trust-extracted\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.492206 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.490282 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-bound-sa-token\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.492206 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.490309 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e099b6bc-addb-428b-9a88-2c8b8e544f46-config\") pod \"service-ca-operator-544d749c8c-96jwb\" (UID: \"e099b6bc-addb-428b-9a88-2c8b8e544f46\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" Nov 03 22:07:01.492206 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.490339 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0a4cedce-72c5-4357-8001-23a5e14d7d9d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-p6mft\" (UID: \"0a4cedce-72c5-4357-8001-23a5e14d7d9d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:01.492206 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.490373 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-trusted-ca\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.492936 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.490400 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9kvp\" (UniqueName: \"kubernetes.io/projected/e099b6bc-addb-428b-9a88-2c8b8e544f46-kube-api-access-b9kvp\") pod \"service-ca-operator-544d749c8c-96jwb\" (UID: \"e099b6bc-addb-428b-9a88-2c8b8e544f46\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" Nov 03 22:07:01.492936 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.490433 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:01.492936 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.490457 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e01c32-cde7-4c5e-add4-ca8c17e82e34-tmp\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.492936 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.490480 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-certificates\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.492936 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.490485 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z\" (UID: \"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" Nov 03 22:07:01.492936 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.490541 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grqtw\" (UniqueName: \"kubernetes.io/projected/2e0d6adb-5887-42af-ad83-48863e1ac386-kube-api-access-grqtw\") pod \"network-check-source-6f78d59f99-q4b9r\" (UID: \"2e0d6adb-5887-42af-ad83-48863e1ac386\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q4b9r" Nov 03 22:07:01.492936 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.490572 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ab2929c7-ab47-4660-80eb-f973bf6efdae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:01.492936 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.490600 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whwxk\" (UniqueName: \"kubernetes.io/projected/ab2929c7-ab47-4660-80eb-f973bf6efdae-kube-api-access-whwxk\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:01.492936 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.490807 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e01c32-cde7-4c5e-add4-ca8c17e82e34-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.492936 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.490920 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:07:01.492936 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.490935 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d66d58db-phhsh: secret "image-registry-tls" not found Nov 03 22:07:01.492936 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.490958 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e01c32-cde7-4c5e-add4-ca8c17e82e34-service-ca-bundle\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.492936 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.490974 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls podName:aacf527d-b30b-48cd-b5fd-be3eba9a2095 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:01.990960295 +0000 UTC m=+32.232229347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls") pod "image-registry-58d66d58db-phhsh" (UID: "aacf527d-b30b-48cd-b5fd-be3eba9a2095") : secret "image-registry-tls" not found Nov 03 22:07:01.492936 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.491030 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:07:01.492936 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.491091 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls podName:43b5a34d-6354-4586-9dd1-55279e0dcfad nodeName:}" failed. No retries permitted until 2026-11-03 22:07:01.991054708 +0000 UTC m=+32.232323756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls") pod "dns-default-rntwm" (UID: "43b5a34d-6354-4586-9dd1-55279e0dcfad") : secret "dns-default-metrics-tls" not found Nov 03 22:07:01.492936 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.491365 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aacf527d-b30b-48cd-b5fd-be3eba9a2095-ca-trust-extracted\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.493759 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.491389 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-image-registry-private-configuration\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.493759 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.491739 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-image-registry-private-configuration\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.493759 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.492015 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60e01c32-cde7-4c5e-add4-ca8c17e82e34-serving-cert\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.493759 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.492112 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e099b6bc-addb-428b-9a88-2c8b8e544f46-config\") pod \"service-ca-operator-544d749c8c-96jwb\" (UID: \"e099b6bc-addb-428b-9a88-2c8b8e544f46\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" Nov 03 22:07:01.493759 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.492121 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:07:01.493759 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.492161 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls podName:f2a09669-a5d7-4dbb-9ddf-842f4f919028 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:01.992147977 +0000 UTC m=+32.233416995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dfffx" (UID: "f2a09669-a5d7-4dbb-9ddf-842f4f919028") : secret "samples-operator-tls" not found Nov 03 22:07:01.493759 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.492161 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:01.493759 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.492369 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e01c32-cde7-4c5e-add4-ca8c17e82e34-tmp\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.493759 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.492775 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c0ac5b4-308e-4eca-8301-2c470cecdbef-ca-trust-extracted\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.493759 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.492798 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0a4cedce-72c5-4357-8001-23a5e14d7d9d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-p6mft\" (UID: \"0a4cedce-72c5-4357-8001-23a5e14d7d9d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:01.493759 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.493675 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-trusted-ca\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.493759 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.493731 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43b5a34d-6354-4586-9dd1-55279e0dcfad-config-volume\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:01.494354 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.493808 2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:07:01.494354 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.493867 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert podName:0a4cedce-72c5-4357-8001-23a5e14d7d9d nodeName:}" failed. No retries permitted until 2026-11-03 22:07:01.993851344 +0000 UTC m=+32.235120370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p6mft" (UID: "0a4cedce-72c5-4357-8001-23a5e14d7d9d") : secret "networking-console-plugin-cert" not found Nov 03 22:07:01.494437 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.494345 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ab2929c7-ab47-4660-80eb-f973bf6efdae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:01.494552 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.494529 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3b08229-d363-4336-bb37-89df96b43dbe-cert\") pod \"ingress-canary-w55xl\" (UID: \"d3b08229-d363-4336-bb37-89df96b43dbe\") " pod="openshift-ingress-canary/ingress-canary-w55xl" Nov 03 22:07:01.494666 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.494640 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-trusted-ca\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.494884 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.494865 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z\" (UID: \"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" Nov 03 22:07:01.494948 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.494897 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-installation-pull-secrets\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.494948 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.494892 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1498363-2c98-4046-8feb-abb772a80149-serving-cert\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:01.495082 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.495009 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-default-certificate\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:01.495143 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.495084 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-stats-auth\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:01.496200 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.496179 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e099b6bc-addb-428b-9a88-2c8b8e544f46-serving-cert\") pod \"service-ca-operator-544d749c8c-96jwb\" (UID: \"e099b6bc-addb-428b-9a88-2c8b8e544f46\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" Nov 03 22:07:01.496280 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.496240 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-installation-pull-secrets\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.532356 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.532314 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-f88sj" Nov 03 22:07:01.551312 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.551276 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-bound-sa-token\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.551485 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.551372 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptsfs\" (UniqueName: \"kubernetes.io/projected/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-kube-api-access-ptsfs\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:01.552453 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.551861 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zt5p\" (UniqueName: \"kubernetes.io/projected/f2a09669-a5d7-4dbb-9ddf-842f4f919028-kube-api-access-8zt5p\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:01.554204 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.554163 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v728z\" (UniqueName: \"kubernetes.io/projected/43929f2c-6b7a-4f25-8a97-51f6bee5481a-kube-api-access-v728z\") pod \"volume-data-source-validator-58c9f5f46d-6nsjz\" (UID: \"43929f2c-6b7a-4f25-8a97-51f6bee5481a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6nsjz" Nov 03 22:07:01.555103 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.555009 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8cv4\" (UniqueName: \"kubernetes.io/projected/43b5a34d-6354-4586-9dd1-55279e0dcfad-kube-api-access-m8cv4\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:01.555484 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.555460 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms2cm\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-kube-api-access-ms2cm\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.555594 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.555552 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-bound-sa-token\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.555730 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.555704 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz8gb\" (UniqueName: \"kubernetes.io/projected/60e01c32-cde7-4c5e-add4-ca8c17e82e34-kube-api-access-kz8gb\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.555982 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.555964 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgntw\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-kube-api-access-qgntw\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.556034 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.555994 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzd9z\" (UniqueName: \"kubernetes.io/projected/508293cb-d657-4cfb-a67a-0155a44d6e47-kube-api-access-nzd9z\") pod \"kas-connection-checker-685895c7c9-z2q8s\" (UID: \"508293cb-d657-4cfb-a67a-0155a44d6e47\") " pod="kube-system/kas-connection-checker-685895c7c9-z2q8s" Nov 03 22:07:01.556437 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.556416 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9kvp\" (UniqueName: \"kubernetes.io/projected/e099b6bc-addb-428b-9a88-2c8b8e544f46-kube-api-access-b9kvp\") pod \"service-ca-operator-544d749c8c-96jwb\" (UID: \"e099b6bc-addb-428b-9a88-2c8b8e544f46\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" Nov 03 22:07:01.557009 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.556964 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whwxk\" (UniqueName: \"kubernetes.io/projected/ab2929c7-ab47-4660-80eb-f973bf6efdae-kube-api-access-whwxk\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:01.557122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.557033 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdh26\" (UniqueName: \"kubernetes.io/projected/ec658817-ab95-4b41-a457-d067b9754e3e-kube-api-access-wdh26\") pod \"kas-connection-checker-685895c7c9-tc9n2\" (UID: \"ec658817-ab95-4b41-a457-d067b9754e3e\") " pod="kube-system/kas-connection-checker-685895c7c9-tc9n2" Nov 03 22:07:01.557122 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.557076 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjj86\" (UniqueName: \"kubernetes.io/projected/d1498363-2c98-4046-8feb-abb772a80149-kube-api-access-tjj86\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:01.557504 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.557485 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grqtw\" (UniqueName: \"kubernetes.io/projected/2e0d6adb-5887-42af-ad83-48863e1ac386-kube-api-access-grqtw\") pod \"network-check-source-6f78d59f99-q4b9r\" (UID: \"2e0d6adb-5887-42af-ad83-48863e1ac386\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q4b9r" Nov 03 22:07:01.558030 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.558006 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf54n\" (UniqueName: \"kubernetes.io/projected/d3b08229-d363-4336-bb37-89df96b43dbe-kube-api-access-gf54n\") pod \"ingress-canary-w55xl\" (UID: \"d3b08229-d363-4336-bb37-89df96b43dbe\") " pod="openshift-ingress-canary/ingress-canary-w55xl" Nov 03 22:07:01.558224 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.558206 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx9rl\" (UniqueName: \"kubernetes.io/projected/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3-kube-api-access-fx9rl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z\" (UID: \"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" Nov 03 22:07:01.564220 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.564157 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6nsjz" Nov 03 22:07:01.571350 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.571315 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:01.587409 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.587374 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" Nov 03 22:07:01.597655 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.597627 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" Nov 03 22:07:01.608584 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.608339 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:01.634566 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.634527 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q4b9r" Nov 03 22:07:01.680538 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.665151 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w55xl" Nov 03 22:07:01.841208 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.841108 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tc9n2" Nov 03 22:07:01.850432 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.850401 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-z2q8s" Nov 03 22:07:01.935143 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.935120 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f88sj"] Nov 03 22:07:01.935331 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:07:01.935304 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0af137_f9ce_4f18_8329_3f7df6a728bb.slice/crio-3122197bd471de20b3e5f09ad239cdbd317900d3f2762b3dae1452925882beaa WatchSource:0}: Error finding container 3122197bd471de20b3e5f09ad239cdbd317900d3f2762b3dae1452925882beaa: Status 404 returned error can't find the container with id 3122197bd471de20b3e5f09ad239cdbd317900d3f2762b3dae1452925882beaa Nov 03 22:07:01.959356 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.959143 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-fqp5x"] Nov 03 22:07:01.969141 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.968033 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6nsjz"] Nov 03 22:07:01.970038 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:07:01.970003 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43929f2c_6b7a_4f25_8a97_51f6bee5481a.slice/crio-8a62620c3bda3a6ce0a7e5d25cafad21779ecb646d07222c0dec3aeeeea7cb33 WatchSource:0}: Error finding container 8a62620c3bda3a6ce0a7e5d25cafad21779ecb646d07222c0dec3aeeeea7cb33: Status 404 returned error can't find the container with id 8a62620c3bda3a6ce0a7e5d25cafad21779ecb646d07222c0dec3aeeeea7cb33 Nov 03 22:07:01.999476 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.999443 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:01.999644 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.999494 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:01.999644 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.999567 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:01.999644 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.999610 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:01.999644 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.999640 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:01.999867 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.999665 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:01.999867 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:01.999690 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:01.999867 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.999611 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:07:01.999867 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.999751 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d66d58db-phhsh: secret "image-registry-tls" not found Nov 03 22:07:01.999867 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.999643 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:07:01.999867 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.999770 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:07:01.999867 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.999667 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:07:01.999867 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.999724 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle podName:b537a9eb-5abf-4f6b-9a35-0f13c5211bf2 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:02.999709767 +0000 UTC m=+33.240978780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle") pod "router-default-84d846b7f5-4qjqn" (UID: "b537a9eb-5abf-4f6b-9a35-0f13c5211bf2") : configmap references non-existent config key: service-ca.crt Nov 03 22:07:01.999867 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.999723 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:07:01.999867 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.999860 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595c4958c9-2rswm: secret "image-registry-tls" not found Nov 03 22:07:01.999867 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.999873 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls podName:aacf527d-b30b-48cd-b5fd-be3eba9a2095 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:02.999855066 +0000 UTC m=+33.241124093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls") pod "image-registry-58d66d58db-phhsh" (UID: "aacf527d-b30b-48cd-b5fd-be3eba9a2095") : secret "image-registry-tls" not found Nov 03 22:07:02.000263 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.999773 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:02.000263 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.999899 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls podName:f2a09669-a5d7-4dbb-9ddf-842f4f919028 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:02.999889042 +0000 UTC m=+33.241158056 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dfffx" (UID: "f2a09669-a5d7-4dbb-9ddf-842f4f919028") : secret "samples-operator-tls" not found Nov 03 22:07:02.000263 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.999917 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls podName:43b5a34d-6354-4586-9dd1-55279e0dcfad nodeName:}" failed. No retries permitted until 2026-11-03 22:07:02.999908137 +0000 UTC m=+33.241177150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls") pod "dns-default-rntwm" (UID: "43b5a34d-6354-4586-9dd1-55279e0dcfad") : secret "dns-default-metrics-tls" not found Nov 03 22:07:02.000263 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.999960 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs podName:b537a9eb-5abf-4f6b-9a35-0f13c5211bf2 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:02.999942561 +0000 UTC m=+33.241211579 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs") pod "router-default-84d846b7f5-4qjqn" (UID: "b537a9eb-5abf-4f6b-9a35-0f13c5211bf2") : secret "router-metrics-certs-default" not found Nov 03 22:07:02.000263 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:01.999993 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls podName:7c0ac5b4-308e-4eca-8301-2c470cecdbef nodeName:}" failed. No retries permitted until 2026-11-03 22:07:02.999986463 +0000 UTC m=+33.241255476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls") pod "image-registry-595c4958c9-2rswm" (UID: "7c0ac5b4-308e-4eca-8301-2c470cecdbef") : secret "image-registry-tls" not found Nov 03 22:07:02.000263 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:02.000005 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls podName:ab2929c7-ab47-4660-80eb-f973bf6efdae nodeName:}" failed. No retries permitted until 2026-11-03 22:07:02.999999911 +0000 UTC m=+33.241268924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8gtxf" (UID: "ab2929c7-ab47-4660-80eb-f973bf6efdae") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:02.000263 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.000028 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p6mft\" (UID: \"0a4cedce-72c5-4357-8001-23a5e14d7d9d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:02.000263 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:02.000124 2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:07:02.000263 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:02.000211 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert podName:0a4cedce-72c5-4357-8001-23a5e14d7d9d nodeName:}" failed. No retries permitted until 2026-11-03 22:07:03.000199769 +0000 UTC m=+33.241468803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p6mft" (UID: "0a4cedce-72c5-4357-8001-23a5e14d7d9d") : secret "networking-console-plugin-cert" not found Nov 03 22:07:02.012204 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.012142 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb"] Nov 03 22:07:02.021264 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:07:02.021232 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode099b6bc_addb_428b_9a88_2c8b8e544f46.slice/crio-f4abf96d9dfb27b62edfad66707b35b4145cf229c03e4c0019d0185bcdc9f08b WatchSource:0}: Error finding container f4abf96d9dfb27b62edfad66707b35b4145cf229c03e4c0019d0185bcdc9f08b: Status 404 returned error can't find the container with id f4abf96d9dfb27b62edfad66707b35b4145cf229c03e4c0019d0185bcdc9f08b Nov 03 22:07:02.038189 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.038160 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z"] Nov 03 22:07:02.039030 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:07:02.039002 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a20614_9869_4b3f_8c5b_ef5ef6e28bd3.slice/crio-5ce32fe76a18108b503bb0f105fe410a146428619e7c5649321a7dbd41da2990 WatchSource:0}: Error finding container 5ce32fe76a18108b503bb0f105fe410a146428619e7c5649321a7dbd41da2990: Status 404 returned error can't find the container with id 5ce32fe76a18108b503bb0f105fe410a146428619e7c5649321a7dbd41da2990 Nov 03 22:07:02.048698 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:07:02.048649 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1498363_2c98_4046_8feb_abb772a80149.slice/crio-9bb74f6b20d1d4f2cc6749f1464b8d90f8915189459d4b2e2a518580b2d43640 WatchSource:0}: Error finding container 9bb74f6b20d1d4f2cc6749f1464b8d90f8915189459d4b2e2a518580b2d43640: Status 404 returned error can't find the container with id 9bb74f6b20d1d4f2cc6749f1464b8d90f8915189459d4b2e2a518580b2d43640 Nov 03 22:07:02.050023 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.049994 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-9mgrk"] Nov 03 22:07:02.058200 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.058162 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-q4b9r"] Nov 03 22:07:02.059978 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:07:02.059909 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0d6adb_5887_42af_ad83_48863e1ac386.slice/crio-d271aaf453f31dd5f0413c455e5f37f2cb5f1e0ad422f92f928b37ccf3e6afbb WatchSource:0}: Error finding container d271aaf453f31dd5f0413c455e5f37f2cb5f1e0ad422f92f928b37ccf3e6afbb: Status 404 returned error can't find the container with id d271aaf453f31dd5f0413c455e5f37f2cb5f1e0ad422f92f928b37ccf3e6afbb Nov 03 22:07:02.072929 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:07:02.072900 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b08229_d363_4336_bb37_89df96b43dbe.slice/crio-7380d5a0041e74845c7a768e3e0c0426aa81f06b2eeae1aefb8d674c127fd684 WatchSource:0}: Error finding container 7380d5a0041e74845c7a768e3e0c0426aa81f06b2eeae1aefb8d674c127fd684: Status 404 returned error can't find the container with id 7380d5a0041e74845c7a768e3e0c0426aa81f06b2eeae1aefb8d674c127fd684 Nov 03 22:07:02.073930 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.073905 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w55xl"] Nov 03 22:07:02.207686 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:07:02.207648 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec658817_ab95_4b41_a457_d067b9754e3e.slice/crio-94df02ea5a247d6508b0d2b021948e6ba7092aaba40d15195fc16e7f8937d064 WatchSource:0}: Error finding container 94df02ea5a247d6508b0d2b021948e6ba7092aaba40d15195fc16e7f8937d064: Status 404 returned error can't find the container with id 94df02ea5a247d6508b0d2b021948e6ba7092aaba40d15195fc16e7f8937d064 Nov 03 22:07:02.208637 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.208608 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tc9n2"] Nov 03 22:07:02.234208 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:07:02.234175 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508293cb_d657_4cfb_a67a_0155a44d6e47.slice/crio-88a437794c540a565f5a3813a90986236efc8972cc883d5895c8e7b38d905f0c WatchSource:0}: Error finding container 88a437794c540a565f5a3813a90986236efc8972cc883d5895c8e7b38d905f0c: Status 404 returned error can't find the container with id 88a437794c540a565f5a3813a90986236efc8972cc883d5895c8e7b38d905f0c Nov 03 22:07:02.234963 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.234938 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-z2q8s"] Nov 03 22:07:02.475490 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.475397 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-z2q8s" event={"ID":"508293cb-d657-4cfb-a67a-0155a44d6e47","Type":"ContainerStarted","Data":"b2b48a1351b633f70dd9b0ac6524c9b9e0d096bb841e84bfcf6e59e7f631ee86"} Nov 03 22:07:02.475490 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.475462 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-z2q8s" event={"ID":"508293cb-d657-4cfb-a67a-0155a44d6e47","Type":"ContainerStarted","Data":"88a437794c540a565f5a3813a90986236efc8972cc883d5895c8e7b38d905f0c"} Nov 03 22:07:02.476696 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.476658 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q4b9r" event={"ID":"2e0d6adb-5887-42af-ad83-48863e1ac386","Type":"ContainerStarted","Data":"d271aaf453f31dd5f0413c455e5f37f2cb5f1e0ad422f92f928b37ccf3e6afbb"} Nov 03 22:07:02.477896 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.477867 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6nsjz" event={"ID":"43929f2c-6b7a-4f25-8a97-51f6bee5481a","Type":"ContainerStarted","Data":"8a62620c3bda3a6ce0a7e5d25cafad21779ecb646d07222c0dec3aeeeea7cb33"} Nov 03 22:07:02.479395 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.479366 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-f88sj" event={"ID":"fe0af137-f9ce-4f18-8329-3f7df6a728bb","Type":"ContainerStarted","Data":"b991478bf790594efbc03fea1de8b7640645aacfec70dc60d2ec8d858e1c584d"} Nov 03 22:07:02.479588 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.479411 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-f88sj" event={"ID":"fe0af137-f9ce-4f18-8329-3f7df6a728bb","Type":"ContainerStarted","Data":"3122197bd471de20b3e5f09ad239cdbd317900d3f2762b3dae1452925882beaa"} Nov 03 22:07:02.480469 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.480435 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w55xl" event={"ID":"d3b08229-d363-4336-bb37-89df96b43dbe","Type":"ContainerStarted","Data":"7380d5a0041e74845c7a768e3e0c0426aa81f06b2eeae1aefb8d674c127fd684"} Nov 03 22:07:02.481577 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.481553 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" event={"ID":"d1498363-2c98-4046-8feb-abb772a80149","Type":"ContainerStarted","Data":"9bb74f6b20d1d4f2cc6749f1464b8d90f8915189459d4b2e2a518580b2d43640"} Nov 03 22:07:02.483052 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.483022 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tc9n2" event={"ID":"ec658817-ab95-4b41-a457-d067b9754e3e","Type":"ContainerStarted","Data":"e7974d8d561191c49f1963b21fca981394b4d5642a8b9e518a8090204d533a7d"} Nov 03 22:07:02.483179 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.483089 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tc9n2" event={"ID":"ec658817-ab95-4b41-a457-d067b9754e3e","Type":"ContainerStarted","Data":"94df02ea5a247d6508b0d2b021948e6ba7092aaba40d15195fc16e7f8937d064"} Nov 03 22:07:02.484278 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.484251 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" event={"ID":"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3","Type":"ContainerStarted","Data":"5ce32fe76a18108b503bb0f105fe410a146428619e7c5649321a7dbd41da2990"} Nov 03 22:07:02.485594 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.485543 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" event={"ID":"e099b6bc-addb-428b-9a88-2c8b8e544f46","Type":"ContainerStarted","Data":"f4abf96d9dfb27b62edfad66707b35b4145cf229c03e4c0019d0185bcdc9f08b"} Nov 03 22:07:02.487138 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:02.487104 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fqp5x" event={"ID":"60e01c32-cde7-4c5e-add4-ca8c17e82e34","Type":"ContainerStarted","Data":"71703b5d5660eec90c226aae448017a584936bdd089dc9aea121c8176c1aef3f"} Nov 03 22:07:03.010427 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.010385 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:03.010628 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.010443 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:03.010628 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.010490 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:03.010628 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.010516 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:03.010628 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.010559 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:07:03.010628 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.010563 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:03.010828 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.010623 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:07:03.010828 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.010650 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595c4958c9-2rswm: secret "image-registry-tls" not found Nov 03 22:07:03.010828 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.010637 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs podName:b537a9eb-5abf-4f6b-9a35-0f13c5211bf2 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:05.01061645 +0000 UTC m=+35.251885486 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs") pod "router-default-84d846b7f5-4qjqn" (UID: "b537a9eb-5abf-4f6b-9a35-0f13c5211bf2") : secret "router-metrics-certs-default" not found Nov 03 22:07:03.010828 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.010634 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:07:03.010828 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.010637 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:03.010828 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.010730 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle podName:b537a9eb-5abf-4f6b-9a35-0f13c5211bf2 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:05.010706594 +0000 UTC m=+35.251975614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle") pod "router-default-84d846b7f5-4qjqn" (UID: "b537a9eb-5abf-4f6b-9a35-0f13c5211bf2") : configmap references non-existent config key: service-ca.crt Nov 03 22:07:03.010828 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.010760 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls podName:7c0ac5b4-308e-4eca-8301-2c470cecdbef nodeName:}" failed. No retries permitted until 2026-11-03 22:07:05.0107506 +0000 UTC m=+35.252019618 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls") pod "image-registry-595c4958c9-2rswm" (UID: "7c0ac5b4-308e-4eca-8301-2c470cecdbef") : secret "image-registry-tls" not found Nov 03 22:07:03.010828 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.010781 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls podName:43b5a34d-6354-4586-9dd1-55279e0dcfad nodeName:}" failed. No retries permitted until 2026-11-03 22:07:05.010772342 +0000 UTC m=+35.252041371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls") pod "dns-default-rntwm" (UID: "43b5a34d-6354-4586-9dd1-55279e0dcfad") : secret "dns-default-metrics-tls" not found Nov 03 22:07:03.010828 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.010803 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls podName:ab2929c7-ab47-4660-80eb-f973bf6efdae nodeName:}" failed. No retries permitted until 2026-11-03 22:07:05.010793554 +0000 UTC m=+35.252062574 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8gtxf" (UID: "ab2929c7-ab47-4660-80eb-f973bf6efdae") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:03.011144 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.010848 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p6mft\" (UID: \"0a4cedce-72c5-4357-8001-23a5e14d7d9d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:03.011144 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.010885 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:03.011144 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.010924 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:03.011144 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.010966 2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:07:03.011144 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.011000 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:07:03.011144 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.011017 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d66d58db-phhsh: secret "image-registry-tls" not found Nov 03 22:07:03.011144 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.011026 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:07:03.011144 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.011008 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert podName:0a4cedce-72c5-4357-8001-23a5e14d7d9d nodeName:}" failed. No retries permitted until 2026-11-03 22:07:05.0109957 +0000 UTC m=+35.252264730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p6mft" (UID: "0a4cedce-72c5-4357-8001-23a5e14d7d9d") : secret "networking-console-plugin-cert" not found Nov 03 22:07:03.011144 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.011091 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls podName:aacf527d-b30b-48cd-b5fd-be3eba9a2095 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:05.011057088 +0000 UTC m=+35.252326103 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls") pod "image-registry-58d66d58db-phhsh" (UID: "aacf527d-b30b-48cd-b5fd-be3eba9a2095") : secret "image-registry-tls" not found Nov 03 22:07:03.011144 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.011112 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls podName:f2a09669-a5d7-4dbb-9ddf-842f4f919028 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:05.011102354 +0000 UTC m=+35.252371366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dfffx" (UID: "f2a09669-a5d7-4dbb-9ddf-842f4f919028") : secret "samples-operator-tls" not found Nov 03 22:07:03.214075 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.213288 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:07:03.214075 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.213346 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:07:03.214075 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.213539 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:07:03.214075 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:03.213605 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs podName:393f28cb-5fc8-4b3e-8852-eebe41b42a2b nodeName:}" failed. No retries permitted until 2026-11-03 22:07:35.213585339 +0000 UTC m=+65.454854357 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs") pod "network-metrics-daemon-sfqbq" (UID: "393f28cb-5fc8-4b3e-8852-eebe41b42a2b") : secret "metrics-daemon-secret" not found Nov 03 22:07:03.222270 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.222179 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:07:03.315578 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.314947 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmppm\" (UniqueName: \"kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm\") pod \"network-check-target-z4ct8\" (UID: \"fab1061b-bf17-477e-bfdf-cb8b1d76b0ff\") " pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:07:03.319099 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.319014 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmppm\" (UniqueName: \"kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm\") pod \"network-check-target-z4ct8\" (UID: \"fab1061b-bf17-477e-bfdf-cb8b1d76b0ff\") " pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:07:03.452271 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.451856 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:07:03.458405 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.458184 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:07:03.499693 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.499635 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-z2q8s" podStartSLOduration=447.499618817 podStartE2EDuration="7m27.499618817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:03.499057876 +0000 UTC m=+33.740326912" watchObservedRunningTime="2026-11-03 22:07:03.499618817 +0000 UTC m=+33.740887852" Nov 03 22:07:03.501266 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.500353 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-tc9n2" podStartSLOduration=447.500339006 podStartE2EDuration="7m27.500339006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:03.383774667 +0000 UTC m=+33.625043703" watchObservedRunningTime="2026-11-03 22:07:03.500339006 +0000 UTC m=+33.741608044" Nov 03 22:07:03.615752 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.615681 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-f88sj" podStartSLOduration=447.615657716 podStartE2EDuration="7m27.615657716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:03.613743817 +0000 UTC m=+33.855012877" watchObservedRunningTime="2026-11-03 22:07:03.615657716 +0000 UTC m=+33.856926752" Nov 03 22:07:03.845560 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.845527 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z4ct8"] Nov 03 22:07:03.854770 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:07:03.854724 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab1061b_bf17_477e_bfdf_cb8b1d76b0ff.slice/crio-d573f0aeb40eaadb1f820381e93f2e7f3d717c29fc76c8a6ccd1b3233471cb4d WatchSource:0}: Error finding container d573f0aeb40eaadb1f820381e93f2e7f3d717c29fc76c8a6ccd1b3233471cb4d: Status 404 returned error can't find the container with id d573f0aeb40eaadb1f820381e93f2e7f3d717c29fc76c8a6ccd1b3233471cb4d Nov 03 22:07:03.897243 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:07:03.897205 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d8e137_a970_441d_9800_d1ce1c8f1f94.slice/crio-eff69f0b0c3dbd71f7d78bf2402b361d78ff93424d80e348d6fcbc51ae573f1a WatchSource:0}: Error finding container eff69f0b0c3dbd71f7d78bf2402b361d78ff93424d80e348d6fcbc51ae573f1a: Status 404 returned error can't find the container with id eff69f0b0c3dbd71f7d78bf2402b361d78ff93424d80e348d6fcbc51ae573f1a Nov 03 22:07:03.897720 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:03.897633 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8wmmt"] Nov 03 22:07:04.505344 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:04.505284 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z4ct8" event={"ID":"fab1061b-bf17-477e-bfdf-cb8b1d76b0ff","Type":"ContainerStarted","Data":"d573f0aeb40eaadb1f820381e93f2e7f3d717c29fc76c8a6ccd1b3233471cb4d"} Nov 03 22:07:04.509248 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:04.509169 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8wmmt" event={"ID":"99d8e137-a970-441d-9800-d1ce1c8f1f94","Type":"ContainerStarted","Data":"eff69f0b0c3dbd71f7d78bf2402b361d78ff93424d80e348d6fcbc51ae573f1a"} Nov 03 22:07:05.032468 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:05.032423 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:05.032781 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:05.032501 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:05.032781 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:05.032535 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:05.032781 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:05.032560 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:05.032781 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:05.032588 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:05.032781 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:05.032624 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p6mft\" (UID: \"0a4cedce-72c5-4357-8001-23a5e14d7d9d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:05.032781 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:05.032665 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:05.032781 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:05.032707 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:05.033194 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:05.032840 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:07:05.033194 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:05.032907 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls podName:f2a09669-a5d7-4dbb-9ddf-842f4f919028 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:09.032886782 +0000 UTC m=+39.274155818 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dfffx" (UID: "f2a09669-a5d7-4dbb-9ddf-842f4f919028") : secret "samples-operator-tls" not found Nov 03 22:07:05.033353 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:05.033332 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:07:05.033427 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:05.033395 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs podName:b537a9eb-5abf-4f6b-9a35-0f13c5211bf2 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:09.033378001 +0000 UTC m=+39.274647031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs") pod "router-default-84d846b7f5-4qjqn" (UID: "b537a9eb-5abf-4f6b-9a35-0f13c5211bf2") : secret "router-metrics-certs-default" not found Nov 03 22:07:05.033529 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:05.033463 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle podName:b537a9eb-5abf-4f6b-9a35-0f13c5211bf2 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:09.033453705 +0000 UTC m=+39.274722718 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle") pod "router-default-84d846b7f5-4qjqn" (UID: "b537a9eb-5abf-4f6b-9a35-0f13c5211bf2") : configmap references non-existent config key: service-ca.crt Nov 03 22:07:05.033529 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:05.033511 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:07:05.033529 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:05.033524 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595c4958c9-2rswm: secret "image-registry-tls" not found Nov 03 22:07:05.033685 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:05.033556 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls podName:7c0ac5b4-308e-4eca-8301-2c470cecdbef nodeName:}" failed. No retries permitted until 2026-11-03 22:07:09.033544369 +0000 UTC m=+39.274813382 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls") pod "image-registry-595c4958c9-2rswm" (UID: "7c0ac5b4-308e-4eca-8301-2c470cecdbef") : secret "image-registry-tls" not found Nov 03 22:07:05.033685 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:05.033612 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:07:05.033685 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:05.033641 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls podName:43b5a34d-6354-4586-9dd1-55279e0dcfad nodeName:}" failed. No retries permitted until 2026-11-03 22:07:09.033631265 +0000 UTC m=+39.274900285 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls") pod "dns-default-rntwm" (UID: "43b5a34d-6354-4586-9dd1-55279e0dcfad") : secret "dns-default-metrics-tls" not found Nov 03 22:07:05.033685 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:05.033687 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:05.033893 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:05.033720 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls podName:ab2929c7-ab47-4660-80eb-f973bf6efdae nodeName:}" failed. No retries permitted until 2026-11-03 22:07:09.033711313 +0000 UTC m=+39.274980327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8gtxf" (UID: "ab2929c7-ab47-4660-80eb-f973bf6efdae") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:05.033893 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:05.033758 2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:07:05.033893 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:05.033780 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert podName:0a4cedce-72c5-4357-8001-23a5e14d7d9d nodeName:}" failed. No retries permitted until 2026-11-03 22:07:09.03377217 +0000 UTC m=+39.275041183 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p6mft" (UID: "0a4cedce-72c5-4357-8001-23a5e14d7d9d") : secret "networking-console-plugin-cert" not found Nov 03 22:07:05.033893 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:05.033839 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:07:05.033893 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:05.033849 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d66d58db-phhsh: secret "image-registry-tls" not found Nov 03 22:07:05.034240 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:05.033910 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls podName:aacf527d-b30b-48cd-b5fd-be3eba9a2095 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:09.03386923 +0000 UTC m=+39.275138246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls") pod "image-registry-58d66d58db-phhsh" (UID: "aacf527d-b30b-48cd-b5fd-be3eba9a2095") : secret "image-registry-tls" not found Nov 03 22:07:05.519219 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:05.519178 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6nsjz" event={"ID":"43929f2c-6b7a-4f25-8a97-51f6bee5481a","Type":"ContainerStarted","Data":"19d51e312f3dd30314aebcaad0cab14db789edc17f66fdcf9d2be8d830d194ba"} Nov 03 22:07:05.633916 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:05.633853 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6nsjz" podStartSLOduration=443.032782971 podStartE2EDuration="7m25.633832474s" totalImagesPullingTime="2.601049503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:05.633605878 +0000 UTC m=+35.874874915" watchObservedRunningTime="2026-11-03 22:07:05.633832474 +0000 UTC m=+35.875101510" Nov 03 22:07:07.532285 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:07.532251 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" event={"ID":"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3","Type":"ContainerStarted","Data":"697b465a38cad9e9a0be32a78985bad7083c2b7d461e4706ae1bb573d45e6a2a"} Nov 03 22:07:07.534623 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:07.534591 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fqp5x" event={"ID":"60e01c32-cde7-4c5e-add4-ca8c17e82e34","Type":"ContainerStarted","Data":"6d8b523fd09bacad32ffe9c4d48457dbd1a2efd39af5487f0eb68a70e6ae2a13"} Nov 03 22:07:07.536650 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:07.536626 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/0.log" Nov 03 22:07:07.536783 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:07.536659 2583 generic.go:356] "Generic (PLEG): container finished" podID="d1498363-2c98-4046-8feb-abb772a80149" containerID="dc097c08f5d666a0376677f56fea3c1f5f81e9d3f6c4462c260d6de72bd8c5e3" exitCode=255 Nov 03 22:07:07.536783 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:07.536683 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" event={"ID":"d1498363-2c98-4046-8feb-abb772a80149","Type":"ContainerDied","Data":"dc097c08f5d666a0376677f56fea3c1f5f81e9d3f6c4462c260d6de72bd8c5e3"} Nov 03 22:07:07.537021 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:07.537006 2583 scope.go:122] "RemoveContainer" containerID="dc097c08f5d666a0376677f56fea3c1f5f81e9d3f6c4462c260d6de72bd8c5e3" Nov 03 22:07:07.783208 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:07.783145 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" podStartSLOduration=440.420047844 podStartE2EDuration="7m25.783122752s" totalImagesPullingTime="5.363074908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:07.648826012 +0000 UTC m=+37.890095049" watchObservedRunningTime="2026-11-03 22:07:07.783122752 +0000 UTC m=+38.024391788" Nov 03 22:07:07.900087 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:07.899411 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-fqp5x" podStartSLOduration=442.753321121 podStartE2EDuration="7m27.899389691s" totalImagesPullingTime="5.14606857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:07.783671277 +0000 UTC m=+38.024940313" watchObservedRunningTime="2026-11-03 22:07:07.899389691 +0000 UTC m=+38.140658728" Nov 03 22:07:08.464902 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.464863 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vsnwk"] Nov 03 22:07:08.469156 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.469124 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:08.523100 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.522388 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:08.523100 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.522453 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:08.523100 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.522583 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27hj2\" (UniqueName: \"kubernetes.io/projected/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-api-access-27hj2\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:08.523100 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.522617 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vsnwk"] Nov 03 22:07:08.523100 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.522660 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c059f8a-ed76-4ffb-9f24-be79941af384-crio-socket\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:08.523100 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.522683 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c059f8a-ed76-4ffb-9f24-be79941af384-data-volume\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:08.548033 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.547996 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:07:08.549083 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.548713 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/0.log" Nov 03 22:07:08.549083 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.548764 2583 generic.go:356] "Generic (PLEG): container finished" podID="d1498363-2c98-4046-8feb-abb772a80149" containerID="6aa20fda0a2c8c0f59dd6e4890d8a0a2cd138f508306fdbc9ad380fa430cb880" exitCode=255 Nov 03 22:07:08.549083 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.548878 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" event={"ID":"d1498363-2c98-4046-8feb-abb772a80149","Type":"ContainerDied","Data":"6aa20fda0a2c8c0f59dd6e4890d8a0a2cd138f508306fdbc9ad380fa430cb880"} Nov 03 22:07:08.549083 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.548951 2583 scope.go:122] "RemoveContainer" containerID="dc097c08f5d666a0376677f56fea3c1f5f81e9d3f6c4462c260d6de72bd8c5e3" Nov 03 22:07:08.549367 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.549200 2583 scope.go:122] "RemoveContainer" containerID="6aa20fda0a2c8c0f59dd6e4890d8a0a2cd138f508306fdbc9ad380fa430cb880" Nov 03 22:07:08.549529 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:08.549504 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9mgrk_openshift-console-operator(d1498363-2c98-4046-8feb-abb772a80149)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" podUID="d1498363-2c98-4046-8feb-abb772a80149" Nov 03 22:07:08.557279 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.557250 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" event={"ID":"e099b6bc-addb-428b-9a88-2c8b8e544f46","Type":"ContainerStarted","Data":"142d34367b57e97e74b5a76ee1fdd261c9a9aa7b2f000d7937fdbb0ceb3d7eb8"} Nov 03 22:07:08.573057 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.572160 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbwbg\"" Nov 03 22:07:08.573057 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.572951 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 22:07:08.573057 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.573027 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 22:07:08.624184 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.624148 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c059f8a-ed76-4ffb-9f24-be79941af384-crio-socket\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:08.624184 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.624188 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c059f8a-ed76-4ffb-9f24-be79941af384-data-volume\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:08.624448 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.624360 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:08.624448 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.624413 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:08.624572 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.624555 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27hj2\" (UniqueName: \"kubernetes.io/projected/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-api-access-27hj2\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:08.625009 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.624985 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c059f8a-ed76-4ffb-9f24-be79941af384-crio-socket\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:08.626105 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:08.625697 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:07:08.626105 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:08.625767 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls podName:9c059f8a-ed76-4ffb-9f24-be79941af384 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:09.125745652 +0000 UTC m=+39.367014670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vsnwk" (UID: "9c059f8a-ed76-4ffb-9f24-be79941af384") : secret "insights-runtime-extractor-tls" not found Nov 03 22:07:08.626105 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:08.625803 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:07:08.626105 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:08.625893 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm podName:9c059f8a-ed76-4ffb-9f24-be79941af384 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:09.125873627 +0000 UTC m=+39.367142661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vsnwk" (UID: "9c059f8a-ed76-4ffb-9f24-be79941af384") : configmap "kube-rbac-proxy" not found Nov 03 22:07:08.626529 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.626508 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c059f8a-ed76-4ffb-9f24-be79941af384-data-volume\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:08.688834 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.688753 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27hj2\" (UniqueName: \"kubernetes.io/projected/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-api-access-27hj2\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:08.713715 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.712527 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-wtfch-phz6j"] Nov 03 22:07:08.719543 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.719468 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:08.725650 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.725618 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzpgz\" (UniqueName: \"kubernetes.io/projected/19c22c7c-c7e9-4ef8-9e55-245393662f36-kube-api-access-wzpgz\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:08.725806 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.725722 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/19c22c7c-c7e9-4ef8-9e55-245393662f36-archives-path\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:08.725806 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.725754 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-trusted-ca-bundle\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:08.725806 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.725791 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-service-ca-bundle\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:08.769253 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.769211 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wtfch-phz6j"] Nov 03 22:07:08.772995 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.772965 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cqv8d\"" Nov 03 22:07:08.826542 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.826504 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-service-ca-bundle\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:08.826729 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.826660 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzpgz\" (UniqueName: \"kubernetes.io/projected/19c22c7c-c7e9-4ef8-9e55-245393662f36-kube-api-access-wzpgz\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:08.826729 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.826717 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/19c22c7c-c7e9-4ef8-9e55-245393662f36-archives-path\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:08.826824 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.826735 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-trusted-ca-bundle\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:08.827241 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.827209 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-service-ca-bundle\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:08.827498 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.827424 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/19c22c7c-c7e9-4ef8-9e55-245393662f36-archives-path\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:08.827665 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.827515 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-trusted-ca-bundle\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:08.871146 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.871092 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" podStartSLOduration=420.349303186 podStartE2EDuration="7m5.871056716s" totalImagesPullingTime="5.52175353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:08.87037893 +0000 UTC m=+39.111647965" watchObservedRunningTime="2026-11-03 22:07:08.871056716 +0000 UTC m=+39.112325750" Nov 03 22:07:08.886131 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:08.886096 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzpgz\" (UniqueName: \"kubernetes.io/projected/19c22c7c-c7e9-4ef8-9e55-245393662f36-kube-api-access-wzpgz\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:09.034733 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.034688 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:09.034929 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.034800 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:09.034929 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.034866 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:09.034929 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.034896 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:09.034929 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.034927 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:09.035129 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.034954 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:09.035129 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.034987 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p6mft\" (UID: \"0a4cedce-72c5-4357-8001-23a5e14d7d9d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:09.035129 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.035021 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:09.035265 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.035192 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:07:09.035265 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.035211 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d66d58db-phhsh: secret "image-registry-tls" not found Nov 03 22:07:09.035358 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.035269 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls podName:aacf527d-b30b-48cd-b5fd-be3eba9a2095 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:17.035251147 +0000 UTC m=+47.276520180 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls") pod "image-registry-58d66d58db-phhsh" (UID: "aacf527d-b30b-48cd-b5fd-be3eba9a2095") : secret "image-registry-tls" not found Nov 03 22:07:09.035707 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.035687 2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:07:09.035783 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.035753 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls podName:f2a09669-a5d7-4dbb-9ddf-842f4f919028 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:17.03573316 +0000 UTC m=+47.277002193 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dfffx" (UID: "f2a09669-a5d7-4dbb-9ddf-842f4f919028") : secret "samples-operator-tls" not found Nov 03 22:07:09.035861 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.035809 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:07:09.035861 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.035841 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs podName:b537a9eb-5abf-4f6b-9a35-0f13c5211bf2 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:17.035827774 +0000 UTC m=+47.277096800 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs") pod "router-default-84d846b7f5-4qjqn" (UID: "b537a9eb-5abf-4f6b-9a35-0f13c5211bf2") : secret "router-metrics-certs-default" not found Nov 03 22:07:09.035926 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.035908 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle podName:b537a9eb-5abf-4f6b-9a35-0f13c5211bf2 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:17.035893403 +0000 UTC m=+47.277162463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle") pod "router-default-84d846b7f5-4qjqn" (UID: "b537a9eb-5abf-4f6b-9a35-0f13c5211bf2") : configmap references non-existent config key: service-ca.crt Nov 03 22:07:09.035989 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.035972 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:07:09.035989 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.035983 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595c4958c9-2rswm: secret "image-registry-tls" not found Nov 03 22:07:09.036109 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.036014 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls podName:7c0ac5b4-308e-4eca-8301-2c470cecdbef nodeName:}" failed. No retries permitted until 2026-11-03 22:07:17.036005459 +0000 UTC m=+47.277274487 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls") pod "image-registry-595c4958c9-2rswm" (UID: "7c0ac5b4-308e-4eca-8301-2c470cecdbef") : secret "image-registry-tls" not found Nov 03 22:07:09.036109 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.036082 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:07:09.036222 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.036113 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls podName:43b5a34d-6354-4586-9dd1-55279e0dcfad nodeName:}" failed. No retries permitted until 2026-11-03 22:07:17.036101599 +0000 UTC m=+47.277370627 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls") pod "dns-default-rntwm" (UID: "43b5a34d-6354-4586-9dd1-55279e0dcfad") : secret "dns-default-metrics-tls" not found Nov 03 22:07:09.036222 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.036164 2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:09.036222 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.036192 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls podName:ab2929c7-ab47-4660-80eb-f973bf6efdae nodeName:}" failed. No retries permitted until 2026-11-03 22:07:17.036185664 +0000 UTC m=+47.277454690 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8gtxf" (UID: "ab2929c7-ab47-4660-80eb-f973bf6efdae") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:09.036373 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.036234 2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:07:09.036373 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.036253 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert podName:0a4cedce-72c5-4357-8001-23a5e14d7d9d nodeName:}" failed. No retries permitted until 2026-11-03 22:07:17.036247672 +0000 UTC m=+47.277516697 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p6mft" (UID: "0a4cedce-72c5-4357-8001-23a5e14d7d9d") : secret "networking-console-plugin-cert" not found Nov 03 22:07:09.037526 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.037498 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:09.136531 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.136491 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:09.136707 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.136555 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:09.136853 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.136838 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:07:09.136910 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.136905 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls podName:9c059f8a-ed76-4ffb-9f24-be79941af384 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:10.136886756 +0000 UTC m=+40.378155785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vsnwk" (UID: "9c059f8a-ed76-4ffb-9f24-be79941af384") : secret "insights-runtime-extractor-tls" not found Nov 03 22:07:09.136978 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.136953 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:07:09.137031 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.136983 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm podName:9c059f8a-ed76-4ffb-9f24-be79941af384 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:10.136973425 +0000 UTC m=+40.378242453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vsnwk" (UID: "9c059f8a-ed76-4ffb-9f24-be79941af384") : configmap "kube-rbac-proxy" not found Nov 03 22:07:09.433000 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.432718 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wtfch-phz6j"] Nov 03 22:07:09.543929 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.543884 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gpqdr"] Nov 03 22:07:09.548356 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.548325 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5nc9k"] Nov 03 22:07:09.548928 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.548903 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpqdr" Nov 03 22:07:09.553114 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.553088 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5nc9k" Nov 03 22:07:09.571707 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.571673 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q4b9r" event={"ID":"2e0d6adb-5887-42af-ad83-48863e1ac386","Type":"ContainerStarted","Data":"7aa1a77d2a536025b90908e6b8f63bbbe5feb94aec8158f795dd976f973b2dd1"} Nov 03 22:07:09.573735 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.573675 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w55xl" event={"ID":"d3b08229-d363-4336-bb37-89df96b43dbe","Type":"ContainerStarted","Data":"8cc6bc8f3b8d87a58ee4a8d2283cf9323e03e94faddc5e01c34947e74e7afba6"} Nov 03 22:07:09.580418 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.580378 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:07:09.580844 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.580827 2583 scope.go:122] "RemoveContainer" containerID="6aa20fda0a2c8c0f59dd6e4890d8a0a2cd138f508306fdbc9ad380fa430cb880" Nov 03 22:07:09.581113 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:09.581038 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9mgrk_openshift-console-operator(d1498363-2c98-4046-8feb-abb772a80149)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" podUID="d1498363-2c98-4046-8feb-abb772a80149" Nov 03 22:07:09.583280 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.583236 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wtfch-phz6j" event={"ID":"19c22c7c-c7e9-4ef8-9e55-245393662f36","Type":"ContainerStarted","Data":"f63d3a0316a67bf8071ed5a4c080f5d6f34bc8778614a7f6e6db6515439d7315"} Nov 03 22:07:09.583421 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.583295 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wtfch-phz6j" event={"ID":"19c22c7c-c7e9-4ef8-9e55-245393662f36","Type":"ContainerStarted","Data":"4dbbb34e777a262f3c6c8d1354e17613d9b1829a51059db8c524f77e2fa72386"} Nov 03 22:07:09.598628 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.598585 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gpqdr"] Nov 03 22:07:09.602877 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.602842 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 22:07:09.602877 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.602867 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 22:07:09.603096 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.602951 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-w8n9w\"" Nov 03 22:07:09.603358 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.603338 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5nc9k"] Nov 03 22:07:09.642894 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.642854 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q9b5\" (UniqueName: \"kubernetes.io/projected/fdc0640a-6da9-4cd2-a6a8-07d56a91718e-kube-api-access-9q9b5\") pod \"migrator-66df5d496b-gpqdr\" (UID: \"fdc0640a-6da9-4cd2-a6a8-07d56a91718e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpqdr" Nov 03 22:07:09.643104 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.642921 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6xt\" (UniqueName: \"kubernetes.io/projected/3da2c3b3-3a29-48e6-8281-8bdcd184b3f0-kube-api-access-fh6xt\") pod \"migrator-66df5d496b-5nc9k\" (UID: \"3da2c3b3-3a29-48e6-8281-8bdcd184b3f0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5nc9k" Nov 03 22:07:09.743835 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.743793 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q9b5\" (UniqueName: \"kubernetes.io/projected/fdc0640a-6da9-4cd2-a6a8-07d56a91718e-kube-api-access-9q9b5\") pod \"migrator-66df5d496b-gpqdr\" (UID: \"fdc0640a-6da9-4cd2-a6a8-07d56a91718e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpqdr" Nov 03 22:07:09.743975 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.743857 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6xt\" (UniqueName: \"kubernetes.io/projected/3da2c3b3-3a29-48e6-8281-8bdcd184b3f0-kube-api-access-fh6xt\") pod \"migrator-66df5d496b-5nc9k\" (UID: \"3da2c3b3-3a29-48e6-8281-8bdcd184b3f0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5nc9k" Nov 03 22:07:09.783368 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.782478 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q4b9r" podStartSLOduration=416.511381092 podStartE2EDuration="7m3.782459522s" totalImagesPullingTime="7.27107843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:09.781195926 +0000 UTC m=+40.022464962" watchObservedRunningTime="2026-11-03 22:07:09.782459522 +0000 UTC m=+40.023728556" Nov 03 22:07:09.802766 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.802726 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6xt\" (UniqueName: \"kubernetes.io/projected/3da2c3b3-3a29-48e6-8281-8bdcd184b3f0-kube-api-access-fh6xt\") pod \"migrator-66df5d496b-5nc9k\" (UID: \"3da2c3b3-3a29-48e6-8281-8bdcd184b3f0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5nc9k" Nov 03 22:07:09.802941 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.802730 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q9b5\" (UniqueName: \"kubernetes.io/projected/fdc0640a-6da9-4cd2-a6a8-07d56a91718e-kube-api-access-9q9b5\") pod \"migrator-66df5d496b-gpqdr\" (UID: \"fdc0640a-6da9-4cd2-a6a8-07d56a91718e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpqdr" Nov 03 22:07:09.872483 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.872399 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpqdr" Nov 03 22:07:09.879184 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:09.879151 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5nc9k" Nov 03 22:07:10.138009 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.137894 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-w55xl" podStartSLOduration=2.41829376 podStartE2EDuration="9.137879051s" totalImagesPullingTime="6.719585291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:10.134529424 +0000 UTC m=+40.375798460" watchObservedRunningTime="2026-11-03 22:07:10.137879051 +0000 UTC m=+40.379148086" Nov 03 22:07:10.138189 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.138124 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-n4vt2"] Nov 03 22:07:10.141651 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.141624 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" Nov 03 22:07:10.148539 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.148500 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:10.148705 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:10.148604 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:07:10.148705 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:10.148667 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm podName:9c059f8a-ed76-4ffb-9f24-be79941af384 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:12.148648235 +0000 UTC m=+42.389917287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vsnwk" (UID: "9c059f8a-ed76-4ffb-9f24-be79941af384") : configmap "kube-rbac-proxy" not found Nov 03 22:07:10.150217 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.150188 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:10.150535 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:10.150518 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:07:10.150618 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:10.150585 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls podName:9c059f8a-ed76-4ffb-9f24-be79941af384 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:12.150568309 +0000 UTC m=+42.391837328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vsnwk" (UID: "9c059f8a-ed76-4ffb-9f24-be79941af384") : secret "insights-runtime-extractor-tls" not found Nov 03 22:07:10.177267 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.177203 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-n4vt2"] Nov 03 22:07:10.195253 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.195219 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 22:07:10.195454 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.195322 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 22:07:10.195454 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.195386 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 22:07:10.195454 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.195406 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 22:07:10.195608 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.195505 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-28d69\"" Nov 03 22:07:10.251277 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.251242 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-n4vt2\" (UID: \"46cb82c6-4e9d-49cd-8bc6-0e006d3785ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" Nov 03 22:07:10.251452 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.251369 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad-signing-key\") pod \"service-ca-5fffc97bdf-n4vt2\" (UID: \"46cb82c6-4e9d-49cd-8bc6-0e006d3785ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" Nov 03 22:07:10.251493 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.251481 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdhtw\" (UniqueName: \"kubernetes.io/projected/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad-kube-api-access-gdhtw\") pod \"service-ca-5fffc97bdf-n4vt2\" (UID: \"46cb82c6-4e9d-49cd-8bc6-0e006d3785ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" Nov 03 22:07:10.251558 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.251502 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-wtfch-phz6j" podStartSLOduration=2.251483325 podStartE2EDuration="2.251483325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:10.249984704 +0000 UTC m=+40.491253739" watchObservedRunningTime="2026-11-03 22:07:10.251483325 +0000 UTC m=+40.492752363" Nov 03 22:07:10.264792 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.264755 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gpqdr"] Nov 03 22:07:10.267787 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:07:10.267760 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da2c3b3_3a29_48e6_8281_8bdcd184b3f0.slice/crio-f22caa88c4719e09b9dfa46b83992ca6065011acd2721056b122d1f20fce5f04 WatchSource:0}: Error finding container f22caa88c4719e09b9dfa46b83992ca6065011acd2721056b122d1f20fce5f04: Status 404 returned error can't find the container with id f22caa88c4719e09b9dfa46b83992ca6065011acd2721056b122d1f20fce5f04 Nov 03 22:07:10.267905 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.267857 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5nc9k"] Nov 03 22:07:10.352225 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.352186 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-n4vt2\" (UID: \"46cb82c6-4e9d-49cd-8bc6-0e006d3785ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" Nov 03 22:07:10.352386 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.352285 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad-signing-key\") pod \"service-ca-5fffc97bdf-n4vt2\" (UID: \"46cb82c6-4e9d-49cd-8bc6-0e006d3785ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" Nov 03 22:07:10.352459 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.352432 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdhtw\" (UniqueName: \"kubernetes.io/projected/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad-kube-api-access-gdhtw\") pod \"service-ca-5fffc97bdf-n4vt2\" (UID: \"46cb82c6-4e9d-49cd-8bc6-0e006d3785ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" Nov 03 22:07:10.353039 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.352975 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-n4vt2\" (UID: \"46cb82c6-4e9d-49cd-8bc6-0e006d3785ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" Nov 03 22:07:10.355697 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.355668 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad-signing-key\") pod \"service-ca-5fffc97bdf-n4vt2\" (UID: \"46cb82c6-4e9d-49cd-8bc6-0e006d3785ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" Nov 03 22:07:10.410075 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.409985 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdhtw\" (UniqueName: \"kubernetes.io/projected/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad-kube-api-access-gdhtw\") pod \"service-ca-5fffc97bdf-n4vt2\" (UID: \"46cb82c6-4e9d-49cd-8bc6-0e006d3785ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" Nov 03 22:07:10.459047 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.459008 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" Nov 03 22:07:10.588567 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.588511 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpqdr" event={"ID":"fdc0640a-6da9-4cd2-a6a8-07d56a91718e","Type":"ContainerStarted","Data":"c56e64d2eb1e24c9deffacafe882c562f551be22941192923f84047e5ce12a14"} Nov 03 22:07:10.590424 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.590387 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5nc9k" event={"ID":"3da2c3b3-3a29-48e6-8281-8bdcd184b3f0","Type":"ContainerStarted","Data":"f22caa88c4719e09b9dfa46b83992ca6065011acd2721056b122d1f20fce5f04"} Nov 03 22:07:10.592529 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.592483 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z4ct8" event={"ID":"fab1061b-bf17-477e-bfdf-cb8b1d76b0ff","Type":"ContainerStarted","Data":"43089a64587f477cd962fdd3f39d438cbe89ee162848f6b2dbee0d35c824d62a"} Nov 03 22:07:10.593456 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.593406 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:07:10.707616 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.707513 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z4ct8" podStartSLOduration=34.266073651 podStartE2EDuration="40.707499242s" totalImagesPullingTime="6.441425591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:10.70719564 +0000 UTC m=+40.948464674" watchObservedRunningTime="2026-11-03 22:07:10.707499242 +0000 UTC m=+40.948768332" Nov 03 22:07:10.837810 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:10.837782 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-n4vt2"] Nov 03 22:07:10.840556 ip-10-0-141-84 kubenswrapper[2583]: W1103 22:07:10.840518 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cb82c6_4e9d_49cd_8bc6_0e006d3785ad.slice/crio-9e8442b87c5db678c75b8c3b347ac5abaa513c28c750a4a6f30bfd03c96840ce WatchSource:0}: Error finding container 9e8442b87c5db678c75b8c3b347ac5abaa513c28c750a4a6f30bfd03c96840ce: Status 404 returned error can't find the container with id 9e8442b87c5db678c75b8c3b347ac5abaa513c28c750a4a6f30bfd03c96840ce Nov 03 22:07:11.600935 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:11.600903 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" event={"ID":"46cb82c6-4e9d-49cd-8bc6-0e006d3785ad","Type":"ContainerStarted","Data":"c369feb46d40ab9dfec6826dd764134a408f36db7fad27c4335694c2c21052c7"} Nov 03 22:07:11.601451 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:11.600947 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" event={"ID":"46cb82c6-4e9d-49cd-8bc6-0e006d3785ad","Type":"ContainerStarted","Data":"9e8442b87c5db678c75b8c3b347ac5abaa513c28c750a4a6f30bfd03c96840ce"} Nov 03 22:07:11.609084 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:11.609034 2583 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:11.609473 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:11.609458 2583 scope.go:122] "RemoveContainer" containerID="6aa20fda0a2c8c0f59dd6e4890d8a0a2cd138f508306fdbc9ad380fa430cb880" Nov 03 22:07:11.609696 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:11.609663 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9mgrk_openshift-console-operator(d1498363-2c98-4046-8feb-abb772a80149)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" podUID="d1498363-2c98-4046-8feb-abb772a80149" Nov 03 22:07:11.610073 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:11.610048 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:12.172032 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:12.171987 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:12.172223 ip-10-0-141-84 kubenswrapper[2583]: I1103 22:07:12.172043 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:12.172223 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:12.172145 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:07:12.172223 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:12.172209 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm podName:9c059f8a-ed76-4ffb-9f24-be79941af384 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:16.172194084 +0000 UTC m=+46.413463098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vsnwk" (UID: "9c059f8a-ed76-4ffb-9f24-be79941af384") : configmap "kube-rbac-proxy" not found Nov 03 22:07:12.172350 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:12.172227 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:07:12.172350 ip-10-0-141-84 kubenswrapper[2583]: E1103 22:07:12.172278 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls podName:9c059f8a-ed76-4ffb-9f24-be79941af384 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:16.172261959 +0000 UTC m=+46.413530987 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vsnwk" (UID: "9c059f8a-ed76-4ffb-9f24-be79941af384") : secret "insights-runtime-extractor-tls" not found Nov 03 22:07:12.337444 ip-10-0-141-84 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 22:07:12.355336 ip-10-0-141-84 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 22:07:12.355573 ip-10-0-141-84 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 22:07:12.355642 ip-10-0-141-84 systemd[1]: kubelet.service: Consumed 3.405s CPU time. Nov 03 22:07:12.367676 ip-10-0-141-84 systemd[1]: Starting Kubernetes Kubelet... Nov 03 22:07:12.440995 ip-10-0-141-84 kubenswrapper[5084]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:07:12.440995 ip-10-0-141-84 kubenswrapper[5084]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 22:07:12.440995 ip-10-0-141-84 kubenswrapper[5084]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:07:12.440995 ip-10-0-141-84 kubenswrapper[5084]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:07:12.443482 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443370 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:07:12.443482 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443476 5084 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:07:12.443482 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443482 5084 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:07:12.443482 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443487 5084 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:07:12.443482 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443492 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443497 5084 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443503 5084 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443507 5084 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443511 5084 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443515 5084 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443519 5084 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443523 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443526 5084 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443530 5084 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443534 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443538 5084 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443553 5084 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443558 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443562 5084 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443567 5084 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443571 5084 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443575 5084 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443580 5084 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443583 5084 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:07:12.443822 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443588 5084 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443595 5084 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443602 5084 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443607 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443612 5084 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443616 5084 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443620 5084 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443625 5084 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443629 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443634 5084 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443639 5084 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443647 5084 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443651 5084 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443656 5084 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443662 5084 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443666 5084 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443671 5084 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443675 5084 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443680 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:07:12.444682 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443684 5084 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443689 5084 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443693 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443698 5084 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443702 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443706 5084 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443711 5084 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443715 5084 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443720 5084 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443724 5084 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443728 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443733 5084 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443737 5084 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443742 5084 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443747 5084 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443752 5084 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443757 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443761 5084 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443765 5084 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:07:12.445534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443770 5084 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443774 5084 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443778 5084 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443784 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443789 5084 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443794 5084 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443799 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443804 5084 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443809 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443813 5084 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443817 5084 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443821 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443825 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443830 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443834 5084 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443838 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443842 5084 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443847 5084 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443851 5084 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:07:12.446369 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443856 5084 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443861 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443865 5084 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443869 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443874 5084 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443880 5084 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443885 5084 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443889 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443893 5084 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443897 5084 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443902 5084 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443906 5084 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443911 5084 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443915 5084 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443919 5084 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443923 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443927 5084 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443933 5084 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443937 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:07:12.447251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443942 5084 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443947 5084 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443951 5084 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443956 5084 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443960 5084 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.443965 5084 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445326 5084 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445339 5084 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445344 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445349 5084 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445354 5084 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445359 5084 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445363 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445368 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445373 5084 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445378 5084 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445382 5084 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445386 5084 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445391 5084 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:07:12.448080 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445395 5084 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445402 5084 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445409 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445414 5084 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445419 5084 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445423 5084 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445428 5084 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445433 5084 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445437 5084 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445441 5084 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445446 5084 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445450 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445455 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445460 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445464 5084 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445468 5084 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445473 5084 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445477 5084 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445481 5084 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445485 5084 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:07:12.448929 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445489 5084 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445495 5084 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445499 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445504 5084 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445508 5084 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445512 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445516 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445521 5084 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445525 5084 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445529 5084 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445533 5084 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445537 5084 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445541 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445545 5084 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445549 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445553 5084 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445558 5084 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445562 5084 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445566 5084 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:07:12.449839 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445571 5084 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445575 5084 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445580 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445584 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445588 5084 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445593 5084 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445598 5084 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445603 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445607 5084 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445611 5084 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445616 5084 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445620 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445625 5084 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445629 5084 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445633 5084 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445637 5084 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445641 5084 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445648 5084 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445654 5084 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:07:12.450532 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445664 5084 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445669 5084 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445673 5084 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445677 5084 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445681 5084 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445686 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445699 5084 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445703 5084 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445707 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445712 5084 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445716 5084 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445720 5084 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445725 5084 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445729 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445734 5084 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445738 5084 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445742 5084 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445746 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445751 5084 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445755 5084 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:07:12.451214 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445760 5084 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445765 5084 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445769 5084 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445774 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445778 5084 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445782 5084 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445787 5084 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445791 5084 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445795 5084 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445799 5084 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445804 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445808 5084 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445814 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445818 5084 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.445826 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446731 5084 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446743 5084 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446748 5084 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446752 5084 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:07:12.452170 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446757 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446761 5084 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446766 5084 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446770 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446775 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446780 5084 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446785 5084 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446790 5084 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446794 5084 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446798 5084 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446802 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446807 5084 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446812 5084 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446817 5084 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446822 5084 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446827 5084 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446831 5084 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446836 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446840 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:07:12.452877 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446844 5084 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446849 5084 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446853 5084 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446857 5084 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446861 5084 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446866 5084 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446873 5084 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446878 5084 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446882 5084 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446887 5084 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446891 5084 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446895 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446900 5084 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446904 5084 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446908 5084 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446912 5084 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446917 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446921 5084 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446928 5084 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446933 5084 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:07:12.453683 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446937 5084 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446941 5084 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446945 5084 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446949 5084 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446954 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446958 5084 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446963 5084 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446978 5084 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446983 5084 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446987 5084 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446992 5084 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.446997 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447001 5084 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447005 5084 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447010 5084 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447014 5084 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447019 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447023 5084 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447030 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447035 5084 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:07:12.454554 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447039 5084 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447043 5084 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447048 5084 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447052 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447057 5084 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447077 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447081 5084 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447085 5084 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447089 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447093 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447100 5084 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447105 5084 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447109 5084 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447113 5084 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447120 5084 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447126 5084 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447132 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447137 5084 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447143 5084 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:07:12.455351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447147 5084 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447153 5084 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447159 5084 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447165 5084 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447170 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447174 5084 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447178 5084 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447183 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447187 5084 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447191 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447195 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447202 5084 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447206 5084 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447210 5084 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447214 5084 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447218 5084 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447223 5084 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447227 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447232 5084 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:07:12.456019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447236 5084 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447240 5084 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447244 5084 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447250 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.447256 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447427 5084 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447439 5084 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447449 5084 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447457 5084 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447465 5084 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447470 5084 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447477 5084 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447484 5084 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447489 5084 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447495 5084 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447502 5084 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447508 5084 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447513 5084 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447519 5084 flags.go:64] FLAG: --cgroup-root="" Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447524 5084 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447529 5084 flags.go:64] FLAG: --client-ca-file="" Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447534 5084 flags.go:64] FLAG: --cloud-provider="external" Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447539 5084 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 22:07:12.456870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447547 5084 flags.go:64] FLAG: --cluster-domain="" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447554 5084 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447559 5084 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447565 5084 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447570 5084 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447577 5084 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447583 5084 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447589 5084 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447595 5084 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447600 5084 flags.go:64] FLAG: --contention-profiling="false" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447605 5084 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447610 5084 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447616 5084 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447627 5084 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447634 5084 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447639 5084 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447644 5084 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447649 5084 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447655 5084 flags.go:64] FLAG: --enable-server="true" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447660 5084 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447667 5084 flags.go:64] FLAG: --event-burst="100" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447672 5084 flags.go:64] FLAG: --event-qps="50" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447677 5084 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447682 5084 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 22:07:12.457672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447688 5084 flags.go:64] FLAG: --eviction-hard="" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447696 5084 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447701 5084 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447706 5084 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447712 5084 flags.go:64] FLAG: --eviction-soft="" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447716 5084 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447722 5084 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447727 5084 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447732 5084 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447739 5084 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447743 5084 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447748 5084 flags.go:64] FLAG: --feature-gates="" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447756 5084 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447761 5084 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447766 5084 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447772 5084 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447777 5084 flags.go:64] FLAG: --healthz-port="10248" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447782 5084 flags.go:64] FLAG: --help="false" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447787 5084 flags.go:64] FLAG: --hostname-override="ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447793 5084 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447798 5084 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447805 5084 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447812 5084 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 22:07:12.458571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447818 5084 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447823 5084 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447828 5084 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447833 5084 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447838 5084 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447843 5084 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447848 5084 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447853 5084 flags.go:64] FLAG: --kube-reserved="" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447858 5084 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447863 5084 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447868 5084 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447874 5084 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447879 5084 flags.go:64] FLAG: --lock-file="" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447884 5084 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447889 5084 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447894 5084 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447904 5084 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447909 5084 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447916 5084 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447920 5084 flags.go:64] FLAG: --logging-format="text" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447925 5084 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447932 5084 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447937 5084 flags.go:64] FLAG: --manifest-url="" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447942 5084 flags.go:64] FLAG: --manifest-url-header="" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447950 5084 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 22:07:12.459710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447955 5084 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447961 5084 flags.go:64] FLAG: --max-pods="110" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447966 5084 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447971 5084 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447976 5084 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447983 5084 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447988 5084 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447993 5084 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.447998 5084 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448010 5084 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448015 5084 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448020 5084 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448025 5084 flags.go:64] FLAG: --pod-cidr="" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448030 5084 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448035 5084 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448041 5084 flags.go:64] FLAG: --pods-per-core="0" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448045 5084 flags.go:64] FLAG: --port="10250" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448050 5084 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448055 5084 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-03bb5b8a81e465a71" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448077 5084 flags.go:64] FLAG: --qos-reserved="" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448082 5084 flags.go:64] FLAG: --read-only-port="10255" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448087 5084 flags.go:64] FLAG: --register-node="true" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448092 5084 flags.go:64] FLAG: --register-with-taints="" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448098 5084 flags.go:64] FLAG: --registry-burst="10" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448103 5084 flags.go:64] FLAG: --registry-qps="5" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448110 5084 flags.go:64] FLAG: --reserved-cpus="" Nov 03 22:07:12.460710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448115 5084 flags.go:64] FLAG: --reserved-memory="" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448127 5084 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448132 5084 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448138 5084 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448142 5084 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448147 5084 flags.go:64] FLAG: --runonce="false" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448152 5084 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448158 5084 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448163 5084 flags.go:64] FLAG: --seccomp-default="false" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448168 5084 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448173 5084 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448181 5084 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448186 5084 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448192 5084 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448196 5084 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448201 5084 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448206 5084 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448211 5084 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448216 5084 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448222 5084 flags.go:64] FLAG: --system-cgroups="" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448226 5084 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448237 5084 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448241 5084 flags.go:64] FLAG: --tls-cert-file="" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448246 5084 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448255 5084 flags.go:64] FLAG: --tls-min-version="" Nov 03 22:07:12.461812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448260 5084 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448266 5084 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448271 5084 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448276 5084 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448281 5084 flags.go:64] FLAG: --v="2" Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448289 5084 flags.go:64] FLAG: --version="false" Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448299 5084 flags.go:64] FLAG: --vmodule="" Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448306 5084 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.448313 5084 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448552 5084 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448560 5084 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448565 5084 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448570 5084 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448574 5084 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448580 5084 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448585 5084 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448590 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448594 5084 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448601 5084 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448606 5084 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448611 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448615 5084 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:07:12.462789 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448619 5084 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448624 5084 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448628 5084 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448634 5084 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448638 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448645 5084 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448651 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448657 5084 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448662 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448667 5084 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448671 5084 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448676 5084 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448682 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448687 5084 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448692 5084 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448696 5084 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448703 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448707 5084 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448712 5084 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448716 5084 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:07:12.463645 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448720 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448725 5084 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448730 5084 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448734 5084 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448739 5084 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448744 5084 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448748 5084 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448753 5084 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448759 5084 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448764 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448768 5084 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448773 5084 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448778 5084 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448782 5084 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448787 5084 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448791 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448796 5084 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448800 5084 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448804 5084 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448809 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:07:12.464440 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448814 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448819 5084 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448823 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448827 5084 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448832 5084 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448837 5084 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448842 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448846 5084 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448855 5084 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448859 5084 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448864 5084 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448868 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448872 5084 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448876 5084 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448881 5084 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448885 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448890 5084 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448895 5084 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448899 5084 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:07:12.465135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448906 5084 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448914 5084 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448918 5084 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448923 5084 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448927 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448931 5084 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448935 5084 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448939 5084 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448944 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448948 5084 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448952 5084 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448957 5084 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448961 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448966 5084 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448970 5084 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448975 5084 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448980 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448984 5084 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448989 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:07:12.465606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448994 5084 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:07:12.466406 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.448999 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:07:12.466406 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449003 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:07:12.466406 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449008 5084 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:07:12.466406 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449012 5084 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:07:12.466406 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449017 5084 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:07:12.466406 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449021 5084 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:07:12.466406 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449026 5084 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:07:12.466406 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449030 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:07:12.466406 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449034 5084 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:07:12.466406 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449039 5084 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:07:12.466406 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449043 5084 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:07:12.466406 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449048 5084 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:07:12.466406 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449053 5084 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:07:12.466406 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449076 5084 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:07:12.466406 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.449129 5084 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:07:12.466406 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449200 5084 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449208 5084 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449215 5084 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449221 5084 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449226 5084 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449231 5084 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449235 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449239 5084 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449244 5084 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449248 5084 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449252 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449258 5084 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449263 5084 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449267 5084 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449272 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449276 5084 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449281 5084 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449287 5084 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449292 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:07:12.466981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449296 5084 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449300 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449305 5084 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449310 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449315 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449319 5084 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449324 5084 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449328 5084 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449333 5084 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449337 5084 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449342 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449349 5084 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449353 5084 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449358 5084 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449362 5084 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449367 5084 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449372 5084 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449376 5084 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449381 5084 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:07:12.467680 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449385 5084 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449390 5084 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449394 5084 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449399 5084 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449403 5084 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449407 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449412 5084 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449416 5084 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449420 5084 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449425 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449429 5084 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449434 5084 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449440 5084 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449444 5084 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449449 5084 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449453 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449460 5084 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449467 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449473 5084 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:07:12.468386 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449477 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449482 5084 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449486 5084 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449493 5084 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449497 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449504 5084 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449508 5084 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449512 5084 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449516 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449521 5084 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449526 5084 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449530 5084 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449535 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449539 5084 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449545 5084 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449550 5084 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449554 5084 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449558 5084 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449563 5084 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449567 5084 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:07:12.468932 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449572 5084 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449576 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449580 5084 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449585 5084 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449589 5084 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449593 5084 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449598 5084 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449602 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449607 5084 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449614 5084 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449619 5084 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449623 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449629 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449633 5084 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449638 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449642 5084 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449646 5084 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449652 5084 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449657 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449662 5084 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:07:12.469606 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449666 5084 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449670 5084 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449675 5084 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449679 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449684 5084 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449697 5084 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449702 5084 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449706 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.449710 5084 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.458598 5084 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.458623 5084 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.458650 5084 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.458660 5084 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458778 5084 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458789 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458795 5084 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458800 5084 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458804 5084 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458810 5084 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458815 5084 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:07:12.470267 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458819 5084 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458823 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458827 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458830 5084 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458833 5084 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458836 5084 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458839 5084 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458846 5084 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458850 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458853 5084 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458855 5084 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458859 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458865 5084 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458870 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458875 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458879 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458883 5084 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458888 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458893 5084 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458897 5084 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:07:12.471115 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458904 5084 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458914 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458917 5084 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458922 5084 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458927 5084 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458932 5084 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458936 5084 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458941 5084 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458946 5084 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458950 5084 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458955 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458960 5084 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458966 5084 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458970 5084 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458974 5084 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458979 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.458983 5084 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459010 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459014 5084 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:07:12.471813 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459019 5084 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459022 5084 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459025 5084 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459028 5084 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459030 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459033 5084 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459036 5084 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459039 5084 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459041 5084 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459045 5084 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459047 5084 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459050 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459053 5084 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459056 5084 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459090 5084 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459095 5084 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459099 5084 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459103 5084 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459107 5084 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:07:12.472355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459111 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459118 5084 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459137 5084 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459143 5084 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459148 5084 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459152 5084 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459156 5084 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459160 5084 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459165 5084 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459170 5084 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459174 5084 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459178 5084 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459183 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459187 5084 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459202 5084 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459208 5084 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459213 5084 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459218 5084 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459245 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:07:12.472874 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459252 5084 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459257 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459262 5084 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459267 5084 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459272 5084 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459276 5084 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459282 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459288 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459292 5084 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459305 5084 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459310 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459314 5084 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459318 5084 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459323 5084 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459327 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459331 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459334 5084 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459337 5084 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459340 5084 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:07:12.473697 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459347 5084 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:07:12.474462 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459350 5084 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:07:12.474462 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459353 5084 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:07:12.474462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.459438 5084 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:07:12.474462 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459716 5084 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:07:12.474462 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459728 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:07:12.474462 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459734 5084 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:07:12.474462 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459739 5084 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:07:12.474462 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459744 5084 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:07:12.474462 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459752 5084 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:07:12.474462 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459757 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:07:12.474462 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459761 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:07:12.474462 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459766 5084 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:07:12.474462 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459775 5084 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:07:12.474462 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459779 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:07:12.474462 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459784 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:07:12.474462 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459788 5084 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459792 5084 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459796 5084 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459801 5084 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459806 5084 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459810 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459814 5084 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459818 5084 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459822 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459827 5084 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459831 5084 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459835 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459845 5084 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459849 5084 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459853 5084 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459857 5084 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459862 5084 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459866 5084 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459870 5084 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459874 5084 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:07:12.474911 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459878 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459883 5084 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459887 5084 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459891 5084 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459895 5084 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459899 5084 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459912 5084 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459917 5084 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459921 5084 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459926 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459930 5084 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459934 5084 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459939 5084 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459943 5084 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459948 5084 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459952 5084 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459957 5084 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459961 5084 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459965 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459969 5084 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:07:12.475529 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459976 5084 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459989 5084 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459994 5084 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.459998 5084 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460003 5084 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460008 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460012 5084 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460017 5084 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460021 5084 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460025 5084 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460030 5084 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460034 5084 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460038 5084 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460043 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460047 5084 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460051 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460055 5084 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460075 5084 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460081 5084 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460090 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:07:12.476054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460094 5084 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460098 5084 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460106 5084 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460111 5084 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460116 5084 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460121 5084 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460126 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460130 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460134 5084 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460139 5084 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460143 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460147 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460152 5084 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460156 5084 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460160 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460164 5084 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460169 5084 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460173 5084 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460177 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:07:12.476701 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460182 5084 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:07:12.477307 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460186 5084 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:07:12.477307 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460190 5084 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:07:12.477307 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460200 5084 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:07:12.477307 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460205 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:07:12.477307 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460209 5084 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:07:12.477307 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460213 5084 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:07:12.477307 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460218 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:07:12.477307 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460222 5084 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:07:12.477307 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460226 5084 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:07:12.477307 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460231 5084 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:07:12.477307 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460235 5084 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:07:12.477307 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460241 5084 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:07:12.477307 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460246 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:07:12.477307 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460251 5084 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:07:12.477307 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.460314 5084 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460374 5084 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460382 5084 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460387 5084 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460397 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460402 5084 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460407 5084 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460411 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460415 5084 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460420 5084 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460424 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460428 5084 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460432 5084 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460436 5084 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460441 5084 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460445 5084 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460449 5084 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460458 5084 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460463 5084 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460468 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:07:12.477761 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460472 5084 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460476 5084 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460480 5084 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460484 5084 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460489 5084 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460493 5084 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460497 5084 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460501 5084 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460505 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460510 5084 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460522 5084 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460532 5084 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460536 5084 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460541 5084 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460545 5084 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460551 5084 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460555 5084 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460560 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460564 5084 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460570 5084 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:07:12.478430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460576 5084 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460581 5084 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460586 5084 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460591 5084 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460595 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460600 5084 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460604 5084 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460608 5084 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460613 5084 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460617 5084 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460621 5084 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460625 5084 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460630 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460638 5084 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460642 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460646 5084 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460651 5084 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460655 5084 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460661 5084 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460665 5084 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:07:12.478990 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460669 5084 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460677 5084 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460687 5084 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460692 5084 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460702 5084 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460707 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460712 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460716 5084 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460721 5084 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460725 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460729 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460734 5084 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460743 5084 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460747 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460752 5084 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460756 5084 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460760 5084 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460765 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460769 5084 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:07:12.479518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460774 5084 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460778 5084 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460782 5084 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460787 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460791 5084 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460801 5084 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460805 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460809 5084 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460813 5084 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460818 5084 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460822 5084 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460826 5084 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460830 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460834 5084 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460838 5084 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460849 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460854 5084 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460860 5084 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460865 5084 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:07:12.480042 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460870 5084 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:07:12.480794 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460874 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:07:12.480794 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460878 5084 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:07:12.480794 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460883 5084 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:07:12.480794 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460887 5084 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:07:12.480794 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460891 5084 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:07:12.480794 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460896 5084 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:07:12.480794 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460905 5084 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:07:12.480794 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:12.460910 5084 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:07:12.480794 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.461029 5084 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 22:07:12.480794 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.462752 5084 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 22:07:12.480794 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.462858 5084 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 22:07:12.480794 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.463140 5084 server.go:1014] "Starting client certificate rotation" Nov 03 22:07:12.480794 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.463337 5084 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:07:12.480794 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.463573 5084 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:01:30 +0000 UTC" deadline="2028-04-25 20:46:51.999587991 +0000 UTC" Nov 03 22:07:12.480794 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.463682 5084 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12934h39m39.535936501s" Nov 03 22:07:12.480794 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.463844 5084 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:07:12.481239 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.466008 5084 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:07:12.481239 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.468726 5084 log.go:25] "Validated CRI v1 runtime API" Nov 03 22:07:12.481239 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.471458 5084 log.go:25] "Validated CRI v1 image API" Nov 03 22:07:12.481239 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.474114 5084 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 22:07:12.481239 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.476471 5084 fs.go:135] Filesystem UUIDs: map[246b339f-ec48-4924-91f8-ce7e37d43afc:/dev/nvme0n1p3 3b0bbcb0-28bc-487e-9a34-f9fbc7a19a79:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 22:07:12.481408 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.476508 5084 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/13b009c03f31dd8d6e367b6e482b641a88b43eea787167db3e95fec2fe57e5bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/13b009c03f31dd8d6e367b6e482b641a88b43eea787167db3e95fec2fe57e5bb/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1863b3f07d31ef234ca436008cf74eede29968bca9f058fabab4bb6b13679c63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1863b3f07d31ef234ca436008cf74eede29968bca9f058fabab4bb6b13679c63/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3122197bd471de20b3e5f09ad239cdbd317900d3f2762b3dae1452925882beaa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3122197bd471de20b3e5f09ad239cdbd317900d3f2762b3dae1452925882beaa/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/353f4193ce0d8e6456ba82f832354c4d42ef16743649b886194c732eb09d110b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/353f4193ce0d8e6456ba82f832354c4d42ef16743649b886194c732eb09d110b/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4dbbb34e777a262f3c6c8d1354e17613d9b1829a51059db8c524f77e2fa72386/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4dbbb34e777a262f3c6c8d1354e17613d9b1829a51059db8c524f77e2fa72386/userdata/shm major:0 minor:319 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56546b9723ade1aa7fb25fcdd92c1ba84e66ffbdad14ab5738c52195173ff5b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56546b9723ade1aa7fb25fcdd92c1ba84e66ffbdad14ab5738c52195173ff5b5/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ce32fe76a18108b503bb0f105fe410a146428619e7c5649321a7dbd41da2990/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ce32fe76a18108b503bb0f105fe410a146428619e7c5649321a7dbd41da2990/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71703b5d5660eec90c226aae448017a584936bdd089dc9aea121c8176c1aef3f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71703b5d5660eec90c226aae448017a584936bdd089dc9aea121c8176c1aef3f/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7380d5a0041e74845c7a768e3e0c0426aa81f06b2eeae1aefb8d674c127fd684/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7380d5a0041e74845c7a768e3e0c0426aa81f06b2eeae1aefb8d674c127fd684/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88a437794c540a565f5a3813a90986236efc8972cc883d5895c8e7b38d905f0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88a437794c540a565f5a3813a90986236efc8972cc883d5895c8e7b38d905f0c/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a62620c3bda3a6ce0a7e5d25cafad21779ecb646d07222c0dec3aeeeea7cb33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a62620c3bda3a6ce0a7e5d25cafad21779ecb646d07222c0dec3aeeeea7cb33/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94df02ea5a247d6508b0d2b021948e6ba7092aaba40d15195fc16e7f8937d064/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94df02ea5a247d6508b0d2b021948e6ba7092aaba40d15195fc16e7f8937d064/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9bb74f6b20d1d4f2cc6749f1464b8d90f8915189459d4b2e2a518580b2d43640/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9bb74f6b20d1d4f2cc6749f1464b8d90f8915189459d4b2e2a518580b2d43640/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e8442b87c5db678c75b8c3b347ac5abaa513c28c750a4a6f30bfd03c96840ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e8442b87c5db678c75b8c3b347ac5abaa513c28c750a4a6f30bfd03c96840ce/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a614e692f58d8f7e0cf3ec28a1e79ac57d2e7bcb798b87460a03fbfc01c9ca56/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a614e692f58d8f7e0cf3ec28a1e79ac57d2e7bcb798b87460a03fbfc01c9ca56/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8218e7160e06cba3868bb17fb9f2219dbb98163d029a279ee76a91ec027ee86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8218e7160e06cba3868bb17fb9f2219dbb98163d029a279ee76a91ec027ee86/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aedd4fb1ad6446f67ad8b58eccfa93ee65feb8805918783b757bbf5e8ef6244e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aedd4fb1ad6446f67ad8b58eccfa93ee65feb8805918783b757bbf5e8ef6244e/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7df88dc34097524a925047e9118094cbb03377f7ad56e9ef8490c94fd865425/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7df88dc34097524a925047e9118094cbb03377f7ad56e9ef8490c94fd865425/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c56e64d2eb1e24c9deffacafe882c562f551be22941192923f84047e5ce12a14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c56e64d2eb1e24c9deffacafe882c562f551be22941192923f84047e5ce12a14/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d271aaf453f31dd5f0413c455e5f37f2cb5f1e0ad422f92f928b37ccf3e6afbb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d271aaf453f31dd5f0413c455e5f37f2cb5f1e0ad422f92f928b37ccf3e6afbb/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d573f0aeb40eaadb1f820381e93f2e7f3d717c29fc76c8a6ccd1b3233471cb4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d573f0aeb40eaadb1f820381e93f2e7f3d717c29fc76c8a6ccd1b3233471cb4d/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eee432b62ea81e4f68471f2eee8cf21730bd9548455992a6d708e65938d2cc12/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eee432b62ea81e4f68471f2eee8cf21730bd9548455992a6d708e65938d2cc12/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eff69f0b0c3dbd71f7d78bf2402b361d78ff93424d80e348d6fcbc51ae573f1a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eff69f0b0c3dbd71f7d78bf2402b361d78ff93424d80e348d6fcbc51ae573f1a/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f22caa88c4719e09b9dfa46b83992ca6065011acd2721056b122d1f20fce5f04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f22caa88c4719e09b9dfa46b83992ca6065011acd2721056b122d1f20fce5f04/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2bf980b365434d4e4300fb48c5a170bf5982658d90e2c282949e5c638669e08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2bf980b365434d4e4300fb48c5a170bf5982658d90e2c282949e5c638669e08/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4abf96d9dfb27b62edfad66707b35b4145cf229c03e4c0019d0185bcdc9f08b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4abf96d9dfb27b62edfad66707b35b4145cf229c03e4c0019d0185bcdc9f08b/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19a326ff-4502-4590-9c0d-194f8cf14abd/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/19a326ff-4502-4590-9c0d-194f8cf14abd/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19c22c7c-c7e9-4ef8-9e55-245393662f36/volumes/kubernetes.io~projected/kube-api-access-wzpgz:{mountpoint:/var/lib/kubelet/pods/19c22c7c-c7e9-4ef8-9e55-245393662f36/volumes/kubernetes.io~projected/kube-api-access-wzpgz major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d49e0d2-566e-472c-9c2d-c85256d52de4/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1d49e0d2-566e-472c-9c2d-c85256d52de4/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d49e0d2-566e-472c-9c2d-c85256d52de4/volumes/kubernetes.io~projected/kube-api-access-6vt2f:{mountpoint:/var/lib/kubelet/pods/1d49e0d2-566e-472c-9c2d-c85256d52de4/volumes/kubernetes.io~projected/kube-api-access-6vt2f major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d49e0d2-566e-472c-9c2d-c85256d52de4/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1d49e0d2-566e-472c-9c2d-c85256d52de4/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e0d6adb-5887-42af-ad83-48863e1ac386/volumes/kubernetes.io~projected/kube-api-access-grqtw:{mountpoint:/var/lib/kubelet/pods/2e0d6adb-5887-42af-ad83-48863e1ac386/volumes/kubernetes.io~projected/kube-api-access-grqtw major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37b1fdc9-35d8-4bd3-af0c-cea9299192e8/volumes/kubernetes.io~projected/kube-api-access-tvmk9:{mountpoint:/var/lib/kubelet/pods/37b1fdc9-35d8-4bd3-af0c-cea9299192e8/volumes/kubernetes.io~projected/kube-api-access-tvmk9 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/393f28cb-5fc8-4b3e-8852-eebe41b42a2b/volumes/kubernetes.io~projected/kube-api-access-vb6r6:{mountpoint:/var/lib/kubelet/pods/393f28cb-5fc8-4b3e-8852-eebe41b42a2b/volumes/kubernetes.io~projected/kube-api-access-vb6r6 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3da2c3b3-3a29-48e6-8281-8bdcd184b3f0/volumes/kubernetes.io~projected/kube-api-access-fh6xt:{mountpoint:/var/lib/kubelet/pods/3da2c3b3-3a29-48e6-8281-8bdcd184b3f0/volumes/kubernetes.io~projected/kube-api-access-fh6xt major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/412af756-be10-401f-8413-419d66bafaad/volumes/kubernetes.io~projected/kube-api-access-4drgl:{mountpoint:/var/lib/kubelet/pods/412af756-be10-401f-8413-419d66bafaad/volumes/kubernetes.io~projected/kube-api-access-4drgl major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43929f2c-6b7a-4f25-8a97-51f6bee5481a/volumes/kubernetes.io~projected/kube-api-access-v728z:{mountpoint:/var/lib/kubelet/pods/43929f2c-6b7a-4f25-8a97-51f6bee5481a/volumes/kubernetes.io~projected/kube-api-access-v728z major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43b5a34d-6354-4586-9dd1-55279e0dcfad/volumes/kubernetes.io~projected/kube-api-access-m8cv4:{mountpoint:/var/lib/kubelet/pods/43b5a34d-6354-4586-9dd1-55279e0dcfad/volumes/kubernetes.io~projected/kube-api-access-m8cv4 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad/volumes/kubernetes.io~projected/kube-api-access-gdhtw:{mountpoint:/var/lib/kubelet/pods/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad/volumes/kubernetes.io~projected/kube-api-access-gdhtw major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad/volumes/kubernetes.io~secret/signing-key major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/508293cb-d657-4cfb-a67a-0155a44d6e47/volumes/kubernetes.io~projected/kube-api-access-nzd9z:{mountpoint:/var/lib/kubelet/pods/508293cb-d657-4cfb-a67a-0155a44d6e47/volumes/kubernetes.io~projected/kube-api-access-nzd9z major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60e01c32-cde7-4c5e-add4-ca8c17e82e34/volumes/kubernetes.io~projected/kube-api-access-kz8gb:{mountpoint:/var/lib/kubelet/pods/60e01c32-cde7-4c5e-add4-ca8c17e82e34/volumes/kubernetes.io~projected/kube-api-access-kz8gb major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60e01c32-cde7-4c5e-add4-ca8c17e82e34/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/60e01c32-cde7-4c5e-add4-ca8c17e82e34/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c0ac5b4-308e-4eca-8301-2c470cecdbef/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/7c0ac5b4-308e-4eca-8301-2c470cecdbef/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c0ac5b4-308e-4eca-8301-2c470cecdbef/volumes/kubernetes.io~projected/kube-api-access-qgntw:{mountpoint:/var/lib/kubelet/pods/7c0ac5b4-308e-4eca-8301-2c470cecdbef/volumes/kubernetes.io~projected/kube-api-access-qgntw major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c0ac5b4-308e-4eca-8301-2c470cecdbef/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/7c0ac5b4-308e-4eca-8301-2c470cecdbef/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c0ac5b4-308e-4eca-8301-2c470cecdbef/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/7c0ac5b4-308e-4eca-8301-2c470cecdbef/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/866a1e7a-4728-4bd5-a88e-1a97f36f1eac/volumes/kubernetes.io~projected/kube-api-access-sm7gx:{mountpoint:/var/lib/kubelet/pods/866a1e7a-4728-4bd5-a88e-1a97f36f1eac/volumes/kubernetes.io~projected/kube-api-access-sm7gx major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b599076-85db-4b98-8636-485e964a5e59/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8b599076-85db-4b98-8636-485e964a5e59/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b599076-85db-4b98-8636-485e964a5e59/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8b599076-85db-4b98-8636-485e964a5e59/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b599076-85db-4b98-8636-485e964a5e59/volumes/kubernetes.io~projected/kube-api-access-sz5dx:{mountpoint:/var/lib/kubelet/pods/8b599076-85db-4b98-8636-485e964a5e59/volumes/kubernetes.io~projected/kube-api-access-sz5dx major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95246059-e5c1-4b42-9678-a1c8b2f63e80/volumes/kubernetes.io~projected/kube-api-access-lgxtv:{mountpoint:/var/lib/kubelet/pods/95246059-e5c1-4b42-9678-a1c8b2f63e80/volumes/kubernetes.io~projected/kube-api-access-lgxtv major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99d8e137-a970-441d-9800-d1ce1c8f1f94/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/99d8e137-a970-441d-9800-d1ce1c8f1f94/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c059f8a-ed76-4ffb-9f24-be79941af384/volumes/kubernetes.io~projected/kube-api-access-27hj2:{mountpoint:/var/lib/kubelet/pods/9c059f8a-ed76-4ffb-9f24-be79941af384/volumes/kubernetes.io~projected/kube-api-access-27hj2 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aacf527d-b30b-48cd-b5fd-be3eba9a2095/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/aacf527d-b30b-48cd-b5fd-be3eba9a2095/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aacf527d-b30b-48cd-b5fd-be3eba9a2095/volumes/kubernetes.io~projected/kube-api-access-ms2cm:{mountpoint:/var/lib/kubelet/pods/aacf527d-b30b-48cd-b5fd-be3eba9a2095/volumes/kubernetes.io~projected/kube-api-access-ms2cm major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aacf527d-b30b-48cd-b5fd-be3eba9a2095/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/aacf527d-b30b-48cd-b5fd-be3eba9a2095/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aacf527d-b30b-48cd-b5fd-be3eba9a2095/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/aacf527d-b30b-48cd-b5fd-be3eba9a2095/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab2929c7-ab47-4660-80eb-f973bf6efdae/volumes/kubernetes.io~projected/kube-api-access-whwxk:{mountpoint:/var/lib/kubelet/pods/ab2929c7-ab47-4660-80eb-f973bf6efdae/volumes/kubernetes.io~projected/kube-api-access-whwxk major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3e7a098-088b-4e25-ac35-dc531d619b79/volumes/kubernetes.io~projected/kube-api-access-vlwbf:{mountpoint:/var/lib/kubelet/pods/b3e7a098-088b-4e25-ac35-dc531d619b79/volumes/kubernetes.io~projected/kube-api-access-vlwbf major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2/volumes/kubernetes.io~projected/kube-api-access-ptsfs:{mountpoint:/var/lib/kubelet/pods/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2/volumes/kubernetes.io~projected/kube-api-access-ptsfs major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2/volumes/kubernetes.io~secret/default-certificate major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2/volumes/kubernetes.io~secret/stats-auth major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1498363-2c98-4046-8feb-abb772a80149/volumes/kubernetes.io~projected/kube-api-access-tjj86:{mountpoint:/var/lib/kubelet/pods/d1498363-2c98-4046-8feb-abb772a80149/volumes/kubernetes.io~projected/kube-api-access-tjj86 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1498363-2c98-4046-8feb-abb772a80149/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/d1498363-2c98-4046-8feb-abb772a80149/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3b08229-d363-4336-bb37-89df96b43dbe/volumes/kubernetes.io~projected/kube-api-access-gf54n:{mountpoint:/var/lib/kubelet/pods/d3b08229-d363-4336-bb37-89df96b43dbe/volumes/kubernetes.io~projected/kube-api-access-gf54n major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3b08229-d363-4336-bb37-89df96b43dbe/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/d3b08229-d363-4336-bb37-89df96b43dbe/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e099b6bc-addb-428b-9a88-2c8b8e544f46/volumes/kubernetes.io~projected/kube-api-access-b9kvp:{mountpoint:/var/lib/kubelet/pods/e099b6bc-addb-428b-9a88-2c8b8e544f46/volumes/kubernetes.io~projected/kube-api-access-b9kvp major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e099b6bc-addb-428b-9a88-2c8b8e544f46/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e099b6bc-addb-428b-9a88-2c8b8e544f46/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3/volumes/kubernetes.io~projected/kube-api-access-fx9rl:{mountpoint:/var/lib/kubelet/pods/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3/volumes/kubernetes.io~projected/kube-api-access-fx9rl major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec658817-ab95-4b41-a457-d067b9754e3e/volumes/kubernetes.io~projected/kube-api-access-wdh26:{mountpoint:/var/lib/kubelet/pods/ec658817-ab95-4b41-a457-d067b9754e3e/volumes/kubernetes.io~projected/kube-api-access-wdh26 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2a09669-a5d7-4dbb-9ddf-842f4f919028/volumes/kubernetes.io~projected/kube-api-access-8zt5p:{mountpoint:/var/lib/kubelet/pods/f2a09669-a5d7-4dbb-9ddf-842f4f919028/volumes/kubernetes.io~projected/kube-api-access-8zt5p major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4d66c00-b01b-4c14-998a-d84707b4600b/volumes/kubernetes.io~projected/kube-api-access-fn75q:{mountpoint:/var/lib/kubelet/pods/f4d66c00-b01b-4c14-998a-d84707b4600b/volumes/kubernetes.io~projected/kube-api-access-fn75q major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff/volumes/kubernetes.io~projected/kube-api-access-jmppm:{mountpoint:/var/lib/kubelet/pods/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff/volumes/kubernetes.io~projected/kube-api-access-jmppm major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdc0640a-6da9-4cd2-a6a8-07d56a91718e/volumes/kubernetes.io~projected/kube-api-access-9q9b5:{mountpoint:/var/lib/kubelet/pods/fdc0640a-6da9-4cd2-a6a8-07d56a91718e/volumes/kubernetes.io~projected/kube-api-access-9q9b5 major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe0af137-f9ce-4f18-8329-3f7df6a728bb/volumes/kubernetes.io~projected/kube-api-access-bfhnj:{mountpoint:/var/lib/kubelet/pods/fe0af137-f9ce-4f18-8329-3f7df6a728bb/volumes/kubernetes.io~projected/kube-api-access-bfhnj major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/3190fedc0d1b5825515c643613c4998987666cf5af3b0e9c6189561bcd27928b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/708383f12101333efd4be2bbfa946a39f62a88f4930ba03a128fea3a165375c3/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/09ec4d1d11fbdb3105e826c16002961ad0e94cf51ceb94f41401746cb4785227/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/e6231c985a2c7ed122003f2bda6a96da07400881e6f49a3d49a75c38fd41c623/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/79d881966613f1eb02ce77936e916134f97179f32fc37cd6889c91ca7e1e2b8c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3cf72d6135af1c92c1ac19fe338d40ac7e67c5c73171bf6a75120f4a3a5d7b44/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e63343321dca138e33cb2803714b1d958878561c0c678f94e527dfcb005ec417/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ebac59cd7ef0b9684f73f44994664dc45a374e0ea13333f91caed4da24e399a5/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/aae91070977cf1c130c45d07f270e02ce1f4c3d4541ae1cef47c5ea8919b6df7/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/a7a2eed5edacf19700a0b25b635e53623e0ad2ca8708ab11acc884f943fcdb96/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/964c693614a2245cc130d652e37358df98c462e207a0073c7f0e0664e9d8f01d/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/3d68142ec6c5f43bf196b3eb62038f8b98ad14d3b099e6cb8c91ef9b71732f97/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-151:{mountpoint:/var/lib/containers/storage/overlay/d5adb3145d596331ae860dfc9572bda0fce8fe94aff1575ce204274d54589c52/merged major:0 minor:151 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/aaf04812d65d9f3a25110bf4ca8976571e5496b7abfe13b0be0e367db5700817/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/91df62c9def23099f19a41dc919f3c44e9fa7eedb50631f36d1e5c859746c3f4/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/89b41504701402f3f916d72fb8a2e4c80dac4e3ccc79971c2292cb4b3084115a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/426a0f42bdaeab0288ac18782e29ca4e7e09b1ae871d4091ca9d65537eeb37b0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/223148a5702c7531da3adc8a251876a2cea5f7bb0067b6f9a900825a66e17da4/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/54500344b3e8c18755f41f8dd6c59b64fe293b9ac6cabb5f5945082f17d5df25/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/1af4704f4b00ac442973dcb963a44ab09c0aaf759cb43f16a056381e4af083cc/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8ce74566d59d27d6201d821e4401d79bc7e83f68e2ef0138b3c19ac1369e86d6/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/8ee7433d288829b998de8501e9dede811ea9fb68cde46742890068046d98f131/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/96942c3afaec30642c3fd9a1cdb4fed07471b1415b4fdef6061f2c741e2acbda/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/1452da133ab35b9583f5df1a3d31ae13abc50053849cbdbc1043505f02d8a013/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/e8c55559d1f57d166f5b356ff0d80282c2e9b96d7950d781be37bdfa006a5712/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/e71760440821c648744fa652e303f847fa3520b34a8fb770f91dd33fcee5d693/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/e68dfe734d4d52ad37db8760e82d0566aecaa9f6bb56e814defa194e19367215/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/9f4ccc6860f106c21feda289e6503b532471e04b7fa399ef8aac4c93c7ec355e/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/7c5925da46627aa2e97606a9124a45a1c190b65e12905642f7a7d383bdbfc470/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/65c00174b73880afb7f2d3a5bec3d35a9c6f57b75396ab5272632624401feea7/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/1aecf1fdfd444fec7bceab791a3fc4b22f71890e117ac08d3b237fbfe8078850/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/2e957dd707f1a2e89ad6fcadd39ad9551d434b3fc07b37686b6154374a95bbb2/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/4a8e88d9f15f89bc4bee0f77a01a4ce9a375015d91c2e73bf5df7f4944da9b24/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/a0ef03b1905a90195f3a53a5f653947a16b3ec81305e82ba5b1e1d8825e295d9/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/8cea139dacdf28c2a160f1e2c543579f8cdb5d019a60123341ebb87cd9516879/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/cbe4e7a9a33bef7efa89faea708ce421ead3140b2b49e887d1ee25964658238a/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/bdb2e2721be312946a6e7c78b02adfe97b72b01a99086ee31f3039ccf07bac52/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/c8219f99fee030a1d78281d5ed9d28189a87b48d6d8f95fe68f0b3646c251b90/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/0db794c4246fca5fa8b19f629378ec7ff96fc1521994d1b30b93cdfc3923624a/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/eb9021c6ab1ffebad7b06e7d5f074fae15ece771f9d7ca5a5a6d1f4dc0c8431a/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/d6a82b587b0de22dc716ace533c4c0db087d53a64eb18e8cd077630489c98e8e/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/f81c0179a7ea3cf929fcd57ee5d7e55d901e80f02e37c650e6918da03b05c8b5/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/9432f23e33422a375757244c97d60b9aac389260d53374814568eeccd934be92/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/8bfdf2c49133eb8d13c792a2da72954aebe21014a84c3cde929d629b8e53ee27/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/89c0a7acd5e0cef5675e3788ca24049b58907c4e450ec07d3970ca712a62a71e/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/8b87138b3403a36ec3f50f8760ac8356202727448cc04fb79b11f1ae79e13113/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/d4771010d92206b678f990c3aec346ece2ee4e8525cde981d76e0827086e5ca7/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/ab53687b02269a655aa58756d9d4145b382fdb3aeb8403d331e0b905eb6d7b67/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/f3e1af76de3662e23795df59cbf270160bc004b94821fbc500652a39eec64199/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/3bf3118858a1422604617c5ff718bdd914c9d21fdc9ab9c05007bc8b340deaf6/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/9befedc2606c6b1b0f4ad865f652aff682af1ddef75a26f9e2c8ddac977a4cea/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/c69aba481e75ef9fd468a58f74bcb80f2c3b77bf927e563655e941d09b23f550/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/eb1573807f396a620176954509db3bedc9730824a2bcf6c8e0fff2402b3734f0/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/1e974777f6f53679c986a297aea29e52a8d15dff667702031c42ff283bd16298/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/d9b99a5aa22b288fe9489cdad3246feeeac20a7c8b888bad191b43d79d51d3cd/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/55863e2f2d1037c6ab9846b4581d0f55de0999992bb48739d29142b7bf7f5423/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/6773cc1651f95f20677b58f9640e71f7ee7112144bef789c2ccafd56bc1afe48/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/f1c8e3b5541567177c35e9ecab163035ae2235692e3517cd4c6cc9e747b40f05/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/34f35a6e64beab701001b43238f62413f05c023526f6527d86fa7d867914a3a4/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/a2abb1c917914ebb2d55baecde64f408a9723753d0e2f605078111f61a2de803/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/a6e77f39ba70403cd24a4702f7340b183941ad3491691263b3b0b95fc34d7769/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/a1b5f8763e9ec4c086978216ea60b698e56b7b1e8beacd39296b4d137995cae6/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 22:07:12.486977 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.486562 5084 manager.go:217] Machine: {Timestamp:2026-11-03 22:07:12.486055814 +0000 UTC m=+0.080087785 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100584 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d7e2df2a66aeabefad529ebc636cd SystemUUID:ec2d7e2d-f2a6-6aea-befa-d529ebc636cd BootID:723a35da-0579-423d-8c7f-6c86113ef352 Filesystems:[{Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e099b6bc-addb-428b-9a88-2c8b8e544f46/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1863b3f07d31ef234ca436008cf74eede29968bca9f058fabab4bb6b13679c63/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aedd4fb1ad6446f67ad8b58eccfa93ee65feb8805918783b757bbf5e8ef6244e/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d49e0d2-566e-472c-9c2d-c85256d52de4/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aacf527d-b30b-48cd-b5fd-be3eba9a2095/volumes/kubernetes.io~projected/kube-api-access-ms2cm DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c0ac5b4-308e-4eca-8301-2c470cecdbef/volumes/kubernetes.io~projected/kube-api-access-qgntw DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8b599076-85db-4b98-8636-485e964a5e59/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-151 DeviceMajor:0 DeviceMinor:151 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1498363-2c98-4046-8feb-abb772a80149/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/353f4193ce0d8e6456ba82f832354c4d42ef16743649b886194c732eb09d110b/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56546b9723ade1aa7fb25fcdd92c1ba84e66ffbdad14ab5738c52195173ff5b5/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f2a09669-a5d7-4dbb-9ddf-842f4f919028/volumes/kubernetes.io~projected/kube-api-access-8zt5p DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d271aaf453f31dd5f0413c455e5f37f2cb5f1e0ad422f92f928b37ccf3e6afbb/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99d8e137-a970-441d-9800-d1ce1c8f1f94/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:278 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d573f0aeb40eaadb1f820381e93f2e7f3d717c29fc76c8a6ccd1b3233471cb4d/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aacf527d-b30b-48cd-b5fd-be3eba9a2095/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9bb74f6b20d1d4f2cc6749f1464b8d90f8915189459d4b2e2a518580b2d43640/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4dbbb34e777a262f3c6c8d1354e17613d9b1829a51059db8c524f77e2fa72386/userdata/shm DeviceMajor:0 DeviceMinor:319 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aacf527d-b30b-48cd-b5fd-be3eba9a2095/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71703b5d5660eec90c226aae448017a584936bdd089dc9aea121c8176c1aef3f/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94df02ea5a247d6508b0d2b021948e6ba7092aaba40d15195fc16e7f8937d064/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f4d66c00-b01b-4c14-998a-d84707b4600b/volumes/kubernetes.io~projected/kube-api-access-fn75q DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d49e0d2-566e-472c-9c2d-c85256d52de4/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad/volumes/kubernetes.io~projected/kube-api-access-gdhtw DeviceMajor:0 DeviceMinor:356 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f22caa88c4719e09b9dfa46b83992ca6065011acd2721056b122d1f20fce5f04/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8218e7160e06cba3868bb17fb9f2219dbb98163d029a279ee76a91ec027ee86/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b599076-85db-4b98-8636-485e964a5e59/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3122197bd471de20b3e5f09ad239cdbd317900d3f2762b3dae1452925882beaa/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a62620c3bda3a6ce0a7e5d25cafad21779ecb646d07222c0dec3aeeeea7cb33/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:357 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/393f28cb-5fc8-4b3e-8852-eebe41b42a2b/volumes/kubernetes.io~projected/kube-api-access-vb6r6 DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4abf96d9dfb27b62edfad66707b35b4145cf229c03e4c0019d0185bcdc9f08b/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3da2c3b3-3a29-48e6-8281-8bdcd184b3f0/volumes/kubernetes.io~projected/kube-api-access-fh6xt DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fe0af137-f9ce-4f18-8329-3f7df6a728bb/volumes/kubernetes.io~projected/kube-api-access-bfhnj DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d3b08229-d363-4336-bb37-89df96b43dbe/volumes/kubernetes.io~projected/kube-api-access-gf54n DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c059f8a-ed76-4ffb-9f24-be79941af384/volumes/kubernetes.io~projected/kube-api-access-27hj2 DeviceMajor:0 DeviceMinor:310 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43b5a34d-6354-4586-9dd1-55279e0dcfad/volumes/kubernetes.io~projected/kube-api-access-m8cv4 DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88a437794c540a565f5a3813a90986236efc8972cc883d5895c8e7b38d905f0c/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19c22c7c-c7e9-4ef8-9e55-245393662f36/volumes/kubernetes.io~projected/kube-api-access-wzpgz DeviceMajor:0 DeviceMinor:328 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fdc0640a-6da9-4cd2-a6a8-07d56a91718e/volumes/kubernetes.io~projected/kube-api-access-9q9b5 DeviceMajor:0 DeviceMinor:345 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c56e64d2eb1e24c9deffacafe882c562f551be22941192923f84047e5ce12a14/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/412af756-be10-401f-8413-419d66bafaad/volumes/kubernetes.io~projected/kube-api-access-4drgl DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/13b009c03f31dd8d6e367b6e482b641a88b43eea787167db3e95fec2fe57e5bb/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ec658817-ab95-4b41-a457-d067b9754e3e/volumes/kubernetes.io~projected/kube-api-access-wdh26 DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff/volumes/kubernetes.io~projected/kube-api-access-jmppm DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3/volumes/kubernetes.io~projected/kube-api-access-fx9rl DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95246059-e5c1-4b42-9678-a1c8b2f63e80/volumes/kubernetes.io~projected/kube-api-access-lgxtv DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/60e01c32-cde7-4c5e-add4-ca8c17e82e34/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b3e7a098-088b-4e25-ac35-dc531d619b79/volumes/kubernetes.io~projected/kube-api-access-vlwbf DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aacf527d-b30b-48cd-b5fd-be3eba9a2095/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/43929f2c-6b7a-4f25-8a97-51f6bee5481a/volumes/kubernetes.io~projected/kube-api-access-v728z DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b599076-85db-4b98-8636-485e964a5e59/volumes/kubernetes.io~projected/kube-api-access-sz5dx DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a614e692f58d8f7e0cf3ec28a1e79ac57d2e7bcb798b87460a03fbfc01c9ca56/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19a326ff-4502-4590-9c0d-194f8cf14abd/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e099b6bc-addb-428b-9a88-2c8b8e544f46/volumes/kubernetes.io~projected/kube-api-access-b9kvp DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/37b1fdc9-35d8-4bd3-af0c-cea9299192e8/volumes/kubernetes.io~projected/kube-api-access-tvmk9 DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab2929c7-ab47-4660-80eb-f973bf6efdae/volumes/kubernetes.io~projected/kube-api-access-whwxk DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7380d5a0041e74845c7a768e3e0c0426aa81f06b2eeae1aefb8d674c127fd684/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7df88dc34097524a925047e9118094cbb03377f7ad56e9ef8490c94fd865425/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2bf980b365434d4e4300fb48c5a170bf5982658d90e2c282949e5c638669e08/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c0ac5b4-308e-4eca-8301-2c470cecdbef/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2/volumes/kubernetes.io~projected/kube-api-access-ptsfs DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c0ac5b4-308e-4eca-8301-2c470cecdbef/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60e01c32-cde7-4c5e-add4-ca8c17e82e34/volumes/kubernetes.io~projected/kube-api-access-kz8gb DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/866a1e7a-4728-4bd5-a88e-1a97f36f1eac/volumes/kubernetes.io~projected/kube-api-access-sm7gx DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d49e0d2-566e-472c-9c2d-c85256d52de4/volumes/kubernetes.io~projected/kube-api-access-6vt2f DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c0ac5b4-308e-4eca-8301-2c470cecdbef/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d3b08229-d363-4336-bb37-89df96b43dbe/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ce32fe76a18108b503bb0f105fe410a146428619e7c5649321a7dbd41da2990/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eee432b62ea81e4f68471f2eee8cf21730bd9548455992a6d708e65938d2cc12/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eff69f0b0c3dbd71f7d78bf2402b361d78ff93424d80e348d6fcbc51ae573f1a/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e8442b87c5db678c75b8c3b347ac5abaa513c28c750a4a6f30bfd03c96840ce/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/508293cb-d657-4cfb-a67a-0155a44d6e47/volumes/kubernetes.io~projected/kube-api-access-nzd9z DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d1498363-2c98-4046-8feb-abb772a80149/volumes/kubernetes.io~projected/kube-api-access-tjj86 DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2e0d6adb-5887-42af-ad83-48863e1ac386/volumes/kubernetes.io~projected/kube-api-access-grqtw DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:3122197bd471de2 MacAddress:46:aa:91:3b:b1:2f Speed:10000 Mtu:8901} {Name:4dbbb34e777a262 MacAddress:ea:bf:19:cb:ec:06 Speed:10000 Mtu:8901} {Name:5ce32fe76a18108 MacAddress:66:4c:d7:44:cb:0d Speed:10000 Mtu:8901} {Name:71703b5d5660eec MacAddress:ae:7c:19:21:b7:c7 Speed:10000 Mtu:8901} {Name:7380d5a0041e748 MacAddress:d6:bb:05:50:d3:d3 Speed:10000 Mtu:8901} {Name:88a437794c540a5 MacAddress:5a:1b:8f:31:e4:b7 Speed:10000 Mtu:8901} {Name:8a62620c3bda3a6 MacAddress:a2:28:15:fd:fd:49 Speed:10000 Mtu:8901} {Name:94df02ea5a247d6 MacAddress:ca:cb:74:28:3e:de Speed:10000 Mtu:8901} {Name:9bb74f6b20d1d4f MacAddress:42:de:e9:1b:ed:a6 Speed:10000 Mtu:8901} {Name:9e8442b87c5db67 MacAddress:da:bb:e3:27:bf:5e Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f9:64:30:0b Speed:0 Mtu:9001} {Name:br-int MacAddress:ea:2c:e0:96:b7:ef Speed:0 Mtu:8901} {Name:c56e64d2eb1e24c MacAddress:92:27:6b:24:17:f7 Speed:10000 Mtu:8901} {Name:d271aaf453f31dd MacAddress:46:15:2e:25:24:05 Speed:10000 Mtu:8901} {Name:d573f0aeb40eaad MacAddress:32:79:f3:a2:ef:f3 Speed:10000 Mtu:8901} {Name:eff69f0b0c3dbd7 MacAddress:82:56:5a:64:e2:90 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:f9:64:30:0b Speed:0 Mtu:9001} {Name:f22caa88c4719e0 MacAddress:ce:a0:21:af:ce:21 Speed:10000 Mtu:8901} {Name:f4abf96d9dfb27b MacAddress:0a:12:a3:9a:e8:fe Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:52:1a:fb:2a:16:9b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b6:04:26:68:56:21 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 22:07:12.486977 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.486977 5084 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 22:07:12.487459 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.487033 5084 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 22:07:12.487511 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.487475 5084 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 22:07:12.487711 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.487516 5084 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-84.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 22:07:12.487799 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.487715 5084 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 22:07:12.487799 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.487728 5084 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 22:07:12.487799 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.487744 5084 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:07:12.487799 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.487764 5084 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:07:12.487799 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.487796 5084 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 22:07:12.488105 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.488091 5084 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 22:07:12.488257 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.488240 5084 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 22:07:12.488367 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.488291 5084 kubelet.go:493] "Attempting to sync node with API server" Nov 03 22:07:12.488367 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.488308 5084 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 22:07:12.488367 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.488318 5084 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 22:07:12.488367 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.488329 5084 kubelet.go:405] "Adding apiserver pod source" Nov 03 22:07:12.488367 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.488337 5084 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 22:07:12.490014 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.489194 5084 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 22:07:12.490014 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.489213 5084 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:07:12.490185 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.490107 5084 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.492630 5084 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.492647 5084 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.492773 5084 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.492981 5084 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.493010 5084 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.493144 5084 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.493157 5084 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.493167 5084 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.493177 5084 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.493186 5084 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.493194 5084 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.493200 5084 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.493206 5084 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.493213 5084 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.493219 5084 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.493230 5084 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.493241 5084 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.493282 5084 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 22:07:12.493797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.493292 5084 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 22:07:12.497491 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.497468 5084 server.go:1263] "Started kubelet" Nov 03 22:07:12.498165 ip-10-0-141-84 systemd[1]: Started Kubernetes Kubelet. Nov 03 22:07:12.500534 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.500450 5084 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 22:07:12.500712 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.500700 5084 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 22:07:12.500889 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.500824 5084 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 22:07:12.501754 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.501716 5084 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 22:07:12.502196 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.502167 5084 server.go:317] "Adding debug handlers to kubelet server" Nov 03 22:07:12.506304 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.506266 5084 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 22:07:12.509911 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.508340 5084 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 22:07:12.514184 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.510782 5084 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 22:07:12.514184 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.510804 5084 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 22:07:12.514184 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:12.510993 5084 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-84.us-west-2.compute.internal\" not found" Nov 03 22:07:12.514184 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.512194 5084 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 22:07:12.514184 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.512639 5084 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:01:31 +0000 UTC" deadline="2028-07-29 11:02:37.516518304 +0000 UTC" Nov 03 22:07:12.514184 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.512680 5084 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15204h55m25.003842385s" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517279 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e01c32-cde7-4c5e-add4-ca8c17e82e34" volumeName="kubernetes.io/configmap/60e01c32-cde7-4c5e-add4-ca8c17e82e34-trusted-ca-bundle" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517344 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e0d6adb-5887-42af-ad83-48863e1ac386" volumeName="kubernetes.io/projected/2e0d6adb-5887-42af-ad83-48863e1ac386-kube-api-access-grqtw" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517360 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46cb82c6-4e9d-49cd-8bc6-0e006d3785ad" volumeName="kubernetes.io/secret/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad-signing-key" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517381 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e01c32-cde7-4c5e-add4-ca8c17e82e34" volumeName="kubernetes.io/empty-dir/60e01c32-cde7-4c5e-add4-ca8c17e82e34-tmp" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517395 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0ac5b4-308e-4eca-8301-2c470cecdbef" volumeName="kubernetes.io/empty-dir/7c0ac5b4-308e-4eca-8301-2c470cecdbef-ca-trust-extracted" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517413 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fab1061b-bf17-477e-bfdf-cb8b1d76b0ff" volumeName="kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517426 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d49e0d2-566e-472c-9c2d-c85256d52de4" volumeName="kubernetes.io/projected/1d49e0d2-566e-472c-9c2d-c85256d52de4-kube-api-access-6vt2f" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517439 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46cb82c6-4e9d-49cd-8bc6-0e006d3785ad" volumeName="kubernetes.io/configmap/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad-signing-cabundle" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517472 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0ac5b4-308e-4eca-8301-2c470cecdbef" volumeName="kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-kube-api-access-qgntw" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517486 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3e7a098-088b-4e25-ac35-dc531d619b79" volumeName="kubernetes.io/configmap/b3e7a098-088b-4e25-ac35-dc531d619b79-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517503 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2a09669-a5d7-4dbb-9ddf-842f4f919028" volumeName="kubernetes.io/projected/f2a09669-a5d7-4dbb-9ddf-842f4f919028-kube-api-access-8zt5p" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517516 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdc0640a-6da9-4cd2-a6a8-07d56a91718e" volumeName="kubernetes.io/projected/fdc0640a-6da9-4cd2-a6a8-07d56a91718e-kube-api-access-9q9b5" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517534 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3e7a098-088b-4e25-ac35-dc531d619b79" volumeName="kubernetes.io/configmap/b3e7a098-088b-4e25-ac35-dc531d619b79-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517547 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b537a9eb-5abf-4f6b-9a35-0f13c5211bf2" volumeName="kubernetes.io/projected/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-kube-api-access-ptsfs" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517573 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b537a9eb-5abf-4f6b-9a35-0f13c5211bf2" volumeName="kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-default-certificate" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517588 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19a326ff-4502-4590-9c0d-194f8cf14abd" volumeName="kubernetes.io/configmap/19a326ff-4502-4590-9c0d-194f8cf14abd-konnectivity-ca" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517601 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aacf527d-b30b-48cd-b5fd-be3eba9a2095" volumeName="kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-trusted-ca" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517619 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab2929c7-ab47-4660-80eb-f973bf6efdae" volumeName="kubernetes.io/configmap/ab2929c7-ab47-4660-80eb-f973bf6efdae-telemetry-config" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517632 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1498363-2c98-4046-8feb-abb772a80149" volumeName="kubernetes.io/projected/d1498363-2c98-4046-8feb-abb772a80149-kube-api-access-tjj86" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517651 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e099b6bc-addb-428b-9a88-2c8b8e544f46" volumeName="kubernetes.io/projected/e099b6bc-addb-428b-9a88-2c8b8e544f46-kube-api-access-b9kvp" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517664 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d49e0d2-566e-472c-9c2d-c85256d52de4" volumeName="kubernetes.io/configmap/1d49e0d2-566e-472c-9c2d-c85256d52de4-ovnkube-script-lib" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517678 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37b1fdc9-35d8-4bd3-af0c-cea9299192e8" volumeName="kubernetes.io/projected/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-kube-api-access-tvmk9" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517706 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0ac5b4-308e-4eca-8301-2c470cecdbef" volumeName="kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-certificates" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517718 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0ac5b4-308e-4eca-8301-2c470cecdbef" volumeName="kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-image-registry-private-configuration" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517737 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c059f8a-ed76-4ffb-9f24-be79941af384" volumeName="kubernetes.io/empty-dir/9c059f8a-ed76-4ffb-9f24-be79941af384-data-volume" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517752 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3" volumeName="kubernetes.io/secret/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3-serving-cert" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517765 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a4cedce-72c5-4357-8001-23a5e14d7d9d" volumeName="kubernetes.io/configmap/0a4cedce-72c5-4357-8001-23a5e14d7d9d-nginx-conf" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517789 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d49e0d2-566e-472c-9c2d-c85256d52de4" volumeName="kubernetes.io/configmap/1d49e0d2-566e-472c-9c2d-c85256d52de4-env-overrides" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517815 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0ac5b4-308e-4eca-8301-2c470cecdbef" volumeName="kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-bound-sa-token" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517828 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="866a1e7a-4728-4bd5-a88e-1a97f36f1eac" volumeName="kubernetes.io/empty-dir/866a1e7a-4728-4bd5-a88e-1a97f36f1eac-tmp-dir" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517847 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3e7a098-088b-4e25-ac35-dc531d619b79" volumeName="kubernetes.io/configmap/b3e7a098-088b-4e25-ac35-dc531d619b79-cni-binary-copy" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517859 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e099b6bc-addb-428b-9a88-2c8b8e544f46" volumeName="kubernetes.io/secret/e099b6bc-addb-428b-9a88-2c8b8e544f46-serving-cert" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517871 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19c22c7c-c7e9-4ef8-9e55-245393662f36" volumeName="kubernetes.io/projected/19c22c7c-c7e9-4ef8-9e55-245393662f36-kube-api-access-wzpgz" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517891 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3da2c3b3-3a29-48e6-8281-8bdcd184b3f0" volumeName="kubernetes.io/projected/3da2c3b3-3a29-48e6-8281-8bdcd184b3f0-kube-api-access-fh6xt" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517903 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43b5a34d-6354-4586-9dd1-55279e0dcfad" volumeName="kubernetes.io/empty-dir/43b5a34d-6354-4586-9dd1-55279e0dcfad-tmp-dir" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517920 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b599076-85db-4b98-8636-485e964a5e59" volumeName="kubernetes.io/projected/8b599076-85db-4b98-8636-485e964a5e59-kube-api-access-sz5dx" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517932 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab2929c7-ab47-4660-80eb-f973bf6efdae" volumeName="kubernetes.io/projected/ab2929c7-ab47-4660-80eb-f973bf6efdae-kube-api-access-whwxk" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517945 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19c22c7c-c7e9-4ef8-9e55-245393662f36" volumeName="kubernetes.io/empty-dir/19c22c7c-c7e9-4ef8-9e55-245393662f36-archives-path" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517961 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43b5a34d-6354-4586-9dd1-55279e0dcfad" volumeName="kubernetes.io/projected/43b5a34d-6354-4586-9dd1-55279e0dcfad-kube-api-access-m8cv4" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517975 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e01c32-cde7-4c5e-add4-ca8c17e82e34" volumeName="kubernetes.io/empty-dir/60e01c32-cde7-4c5e-add4-ca8c17e82e34-snapshots" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.517993 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e01c32-cde7-4c5e-add4-ca8c17e82e34" volumeName="kubernetes.io/secret/60e01c32-cde7-4c5e-add4-ca8c17e82e34-serving-cert" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518006 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3e7a098-088b-4e25-ac35-dc531d619b79" volumeName="kubernetes.io/projected/b3e7a098-088b-4e25-ac35-dc531d619b79-kube-api-access-vlwbf" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518147 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec658817-ab95-4b41-a457-d067b9754e3e" volumeName="kubernetes.io/projected/ec658817-ab95-4b41-a457-d067b9754e3e-kube-api-access-wdh26" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518462 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="393f28cb-5fc8-4b3e-8852-eebe41b42a2b" volumeName="kubernetes.io/projected/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-kube-api-access-vb6r6" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518502 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b599076-85db-4b98-8636-485e964a5e59" volumeName="kubernetes.io/empty-dir/8b599076-85db-4b98-8636-485e964a5e59-tmp" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518525 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e099b6bc-addb-428b-9a88-2c8b8e544f46" volumeName="kubernetes.io/configmap/e099b6bc-addb-428b-9a88-2c8b8e544f46-config" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518541 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19c22c7c-c7e9-4ef8-9e55-245393662f36" volumeName="kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-service-ca-bundle" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518554 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37b1fdc9-35d8-4bd3-af0c-cea9299192e8" volumeName="kubernetes.io/configmap/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-daemon-config" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518573 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99d8e137-a970-441d-9800-d1ce1c8f1f94" volumeName="kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518589 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aacf527d-b30b-48cd-b5fd-be3eba9a2095" volumeName="kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-installation-pull-secrets" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518607 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1498363-2c98-4046-8feb-abb772a80149" volumeName="kubernetes.io/configmap/d1498363-2c98-4046-8feb-abb772a80149-trusted-ca" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518628 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d49e0d2-566e-472c-9c2d-c85256d52de4" volumeName="kubernetes.io/secret/1d49e0d2-566e-472c-9c2d-c85256d52de4-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518642 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e01c32-cde7-4c5e-add4-ca8c17e82e34" volumeName="kubernetes.io/projected/60e01c32-cde7-4c5e-add4-ca8c17e82e34-kube-api-access-kz8gb" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518661 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0ac5b4-308e-4eca-8301-2c470cecdbef" volumeName="kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-installation-pull-secrets" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518674 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aacf527d-b30b-48cd-b5fd-be3eba9a2095" volumeName="kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-kube-api-access-ms2cm" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518692 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1498363-2c98-4046-8feb-abb772a80149" volumeName="kubernetes.io/configmap/d1498363-2c98-4046-8feb-abb772a80149-config" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518727 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4d66c00-b01b-4c14-998a-d84707b4600b" volumeName="kubernetes.io/configmap/f4d66c00-b01b-4c14-998a-d84707b4600b-iptables-alerter-script" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518741 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4d66c00-b01b-4c14-998a-d84707b4600b" volumeName="kubernetes.io/projected/f4d66c00-b01b-4c14-998a-d84707b4600b-kube-api-access-fn75q" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518760 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19c22c7c-c7e9-4ef8-9e55-245393662f36" volumeName="kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-trusted-ca-bundle" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518773 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46cb82c6-4e9d-49cd-8bc6-0e006d3785ad" volumeName="kubernetes.io/projected/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad-kube-api-access-gdhtw" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518787 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0ac5b4-308e-4eca-8301-2c470cecdbef" volumeName="kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-trusted-ca" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518806 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95246059-e5c1-4b42-9678-a1c8b2f63e80" volumeName="kubernetes.io/projected/95246059-e5c1-4b42-9678-a1c8b2f63e80-kube-api-access-lgxtv" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518820 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aacf527d-b30b-48cd-b5fd-be3eba9a2095" volumeName="kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-bound-sa-token" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518839 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1498363-2c98-4046-8feb-abb772a80149" volumeName="kubernetes.io/secret/d1498363-2c98-4046-8feb-abb772a80149-serving-cert" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518851 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37b1fdc9-35d8-4bd3-af0c-cea9299192e8" volumeName="kubernetes.io/configmap/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-cni-binary-copy" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518869 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="412af756-be10-401f-8413-419d66bafaad" volumeName="kubernetes.io/projected/412af756-be10-401f-8413-419d66bafaad-kube-api-access-4drgl" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518883 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43929f2c-6b7a-4f25-8a97-51f6bee5481a" volumeName="kubernetes.io/projected/43929f2c-6b7a-4f25-8a97-51f6bee5481a-kube-api-access-v728z" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518897 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="866a1e7a-4728-4bd5-a88e-1a97f36f1eac" volumeName="kubernetes.io/projected/866a1e7a-4728-4bd5-a88e-1a97f36f1eac-kube-api-access-sm7gx" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518915 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aacf527d-b30b-48cd-b5fd-be3eba9a2095" volumeName="kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-certificates" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518930 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b537a9eb-5abf-4f6b-9a35-0f13c5211bf2" volumeName="kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-stats-auth" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518954 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3b08229-d363-4336-bb37-89df96b43dbe" volumeName="kubernetes.io/projected/d3b08229-d363-4336-bb37-89df96b43dbe-kube-api-access-gf54n" seLinuxMountContext="" Nov 03 22:07:12.519100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518968 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3b08229-d363-4336-bb37-89df96b43dbe" volumeName="kubernetes.io/secret/d3b08229-d363-4336-bb37-89df96b43dbe-cert" seLinuxMountContext="" Nov 03 22:07:12.522377 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.519188 5084 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 22:07:12.522377 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.519203 5084 factory.go:55] Registering systemd factory Nov 03 22:07:12.522377 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.519221 5084 factory.go:223] Registration of the systemd container factory successfully Nov 03 22:07:12.524155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.518982 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aacf527d-b30b-48cd-b5fd-be3eba9a2095" volumeName="kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-image-registry-private-configuration" seLinuxMountContext="" Nov 03 22:07:12.524155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.523915 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d49e0d2-566e-472c-9c2d-c85256d52de4" volumeName="kubernetes.io/configmap/1d49e0d2-566e-472c-9c2d-c85256d52de4-ovnkube-config" seLinuxMountContext="" Nov 03 22:07:12.524155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.523992 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b599076-85db-4b98-8636-485e964a5e59" volumeName="kubernetes.io/empty-dir/8b599076-85db-4b98-8636-485e964a5e59-etc-tuned" seLinuxMountContext="" Nov 03 22:07:12.524155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.524029 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aacf527d-b30b-48cd-b5fd-be3eba9a2095" volumeName="kubernetes.io/empty-dir/aacf527d-b30b-48cd-b5fd-be3eba9a2095-ca-trust-extracted" seLinuxMountContext="" Nov 03 22:07:12.524155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.524045 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3" volumeName="kubernetes.io/configmap/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3-config" seLinuxMountContext="" Nov 03 22:07:12.524155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.524058 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe0af137-f9ce-4f18-8329-3f7df6a728bb" volumeName="kubernetes.io/projected/fe0af137-f9ce-4f18-8329-3f7df6a728bb-kube-api-access-bfhnj" seLinuxMountContext="" Nov 03 22:07:12.524155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.524098 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19a326ff-4502-4590-9c0d-194f8cf14abd" volumeName="kubernetes.io/secret/19a326ff-4502-4590-9c0d-194f8cf14abd-agent-certs" seLinuxMountContext="" Nov 03 22:07:12.524155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.524113 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43b5a34d-6354-4586-9dd1-55279e0dcfad" volumeName="kubernetes.io/configmap/43b5a34d-6354-4586-9dd1-55279e0dcfad-config-volume" seLinuxMountContext="" Nov 03 22:07:12.524155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.524126 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="508293cb-d657-4cfb-a67a-0155a44d6e47" volumeName="kubernetes.io/projected/508293cb-d657-4cfb-a67a-0155a44d6e47-kube-api-access-nzd9z" seLinuxMountContext="" Nov 03 22:07:12.524656 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.524138 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95246059-e5c1-4b42-9678-a1c8b2f63e80" volumeName="kubernetes.io/configmap/95246059-e5c1-4b42-9678-a1c8b2f63e80-serviceca" seLinuxMountContext="" Nov 03 22:07:12.524656 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.524188 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c059f8a-ed76-4ffb-9f24-be79941af384" volumeName="kubernetes.io/projected/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-api-access-27hj2" seLinuxMountContext="" Nov 03 22:07:12.524656 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.524202 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3" volumeName="kubernetes.io/projected/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3-kube-api-access-fx9rl" seLinuxMountContext="" Nov 03 22:07:12.524656 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.524215 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e01c32-cde7-4c5e-add4-ca8c17e82e34" volumeName="kubernetes.io/configmap/60e01c32-cde7-4c5e-add4-ca8c17e82e34-service-ca-bundle" seLinuxMountContext="" Nov 03 22:07:12.524656 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.524247 5084 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2a09669-a5d7-4dbb-9ddf-842f4f919028" volumeName="kubernetes.io/configmap/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-config" seLinuxMountContext="" Nov 03 22:07:12.524656 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.524260 5084 reconstruct.go:97] "Volume reconstruction finished" Nov 03 22:07:12.524656 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.524268 5084 reconciler.go:29] "Reconciler: start to sync state" Nov 03 22:07:12.530753 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.529406 5084 factory.go:153] Registering CRI-O factory Nov 03 22:07:12.530753 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.529436 5084 factory.go:223] Registration of the crio container factory successfully Nov 03 22:07:12.530753 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.529458 5084 factory.go:103] Registering Raw factory Nov 03 22:07:12.530753 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.529475 5084 manager.go:1199] Started watching for new ooms in manager Nov 03 22:07:12.537088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.533273 5084 manager.go:319] Starting recovery of all containers Nov 03 22:07:12.581822 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.581789 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:07:12.582108 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.582087 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:07:12.611093 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:12.611040 5084 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-84.us-west-2.compute.internal\" not found" Nov 03 22:07:12.614910 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.614708 5084 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 22:07:12.620262 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.619891 5084 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 22:07:12.620262 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.619917 5084 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 22:07:12.620262 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.619937 5084 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 22:07:12.620262 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:12.619988 5084 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 22:07:12.639747 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.639698 5084 generic.go:356] "Generic (PLEG): container finished" podID="36effd34d282541d8cc7db737926b500" containerID="cd671c3474087a46cf1cd4397fe83dcb408ac6d0227be0d35263fb0cf893b4c6" exitCode=0 Nov 03 22:07:12.653018 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.652992 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:07:12.653178 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.653025 5084 generic.go:356] "Generic (PLEG): container finished" podID="d1498363-2c98-4046-8feb-abb772a80149" containerID="6aa20fda0a2c8c0f59dd6e4890d8a0a2cd138f508306fdbc9ad380fa430cb880" exitCode=255 Nov 03 22:07:12.694639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.694601 5084 generic.go:356] "Generic (PLEG): container finished" podID="b3e7a098-088b-4e25-ac35-dc531d619b79" containerID="90780cb57bb2d16c60232303dedb99aa7f663e863d75e85d1a364f686926080f" exitCode=0 Nov 03 22:07:12.694639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.694630 5084 generic.go:356] "Generic (PLEG): container finished" podID="b3e7a098-088b-4e25-ac35-dc531d619b79" containerID="32163c437a1fc40243027f943f77c401a1e5d27b71e53b214601855c272f7230" exitCode=0 Nov 03 22:07:12.694639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.694636 5084 generic.go:356] "Generic (PLEG): container finished" podID="b3e7a098-088b-4e25-ac35-dc531d619b79" containerID="85cb19323313b002fced32c2972cf2b67a0eea1a0147d113b761fcb9e0bf33a3" exitCode=0 Nov 03 22:07:12.694639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.694643 5084 generic.go:356] "Generic (PLEG): container finished" podID="b3e7a098-088b-4e25-ac35-dc531d619b79" containerID="ece83f952cd0ebe944fa341f644eb179f62116b317eda48acf0794b07ce367c7" exitCode=0 Nov 03 22:07:12.694639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.694650 5084 generic.go:356] "Generic (PLEG): container finished" podID="b3e7a098-088b-4e25-ac35-dc531d619b79" containerID="285673180f92493bf67be500a883a8e342236e7c34106cb14543435cf31285c9" exitCode=0 Nov 03 22:07:12.694922 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.694656 5084 generic.go:356] "Generic (PLEG): container finished" podID="b3e7a098-088b-4e25-ac35-dc531d619b79" containerID="032bbbf9cb5d077c1f5e6d4abb4bf65e1a53aebbd10dbd512baf7b62414963f9" exitCode=0 Nov 03 22:07:12.702193 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.702167 5084 manager.go:324] Recovery completed Nov 03 22:07:12.711194 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:12.711165 5084 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-84.us-west-2.compute.internal\" not found" Nov 03 22:07:12.721079 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:12.721028 5084 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 22:07:12.723488 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.723468 5084 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:07:12.725035 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.725007 5084 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:07:12.725180 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.725050 5084 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:07:12.725180 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.725077 5084 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:07:12.727561 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.727536 5084 cpu_manager.go:226] "Starting" policy="none" Nov 03 22:07:12.727561 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.727553 5084 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 22:07:12.727688 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.727578 5084 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 22:07:12.727688 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.727573 5084 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:07:12.727770 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.727718 5084 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 22:07:12.727770 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.727729 5084 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 22:07:12.727770 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.727748 5084 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 22:07:12.727770 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.727754 5084 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 22:07:12.727770 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.727759 5084 policy_none.go:50] "Start" Nov 03 22:07:12.727770 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.727767 5084 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 22:07:12.728010 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.727777 5084 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 22:07:12.729389 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.729366 5084 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 22:07:12.729457 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.729392 5084 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 22:07:12.729457 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.729399 5084 policy_none.go:44] "Start" Nov 03 22:07:12.733050 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.733026 5084 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:07:12.733200 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.733088 5084 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:07:12.733200 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.733106 5084 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:07:12.734652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.734621 5084 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 22:07:12.740263 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.740235 5084 manager.go:341] "Starting Device Plugin manager" Nov 03 22:07:12.740389 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:12.740277 5084 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 22:07:12.740389 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.740292 5084 server.go:87] "Starting device plugin registration server" Nov 03 22:07:12.740670 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.740649 5084 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 22:07:12.740728 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.740670 5084 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 22:07:12.740811 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.740798 5084 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 22:07:12.740923 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.740891 5084 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 22:07:12.740972 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.740933 5084 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 22:07:12.740972 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.740939 5084 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 22:07:12.741881 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.741795 5084 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:07:12.74092188Z","UUID":"53f7c653-8fd8-48f2-a02b-8806ff6ba16e","Handler":null,"Name":"","Endpoint":""} Nov 03 22:07:12.743506 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:12.743475 5084 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 22:07:12.743623 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:12.743543 5084 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-84.us-west-2.compute.internal\" not found" Nov 03 22:07:12.743969 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.743943 5084 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 22:07:12.744082 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.743988 5084 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 22:07:12.754697 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.754670 5084 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 22:07:12.755197 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.755099 5084 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:07:12.759166 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.759139 5084 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:07:12.759386 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.759356 5084 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:07:12.759574 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.759510 5084 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:07:12.844080 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.844029 5084 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:07:12.845614 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.845584 5084 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 22:07:12.845763 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.845631 5084 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 22:07:12.845763 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.845648 5084 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-84.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 22:07:12.845763 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.845714 5084 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:12.922188 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.922128 5084 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal"] Nov 03 22:07:12.922997 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.922972 5084 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:12.923150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.922986 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" event={"ID":"36effd34d282541d8cc7db737926b500","Type":"ContainerStarted","Data":"bcb3c40cf4180a2b6c7ba34cf7ba4b554c781a49afd687acdd0a60c89fc282ca"} Nov 03 22:07:12.923150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.923099 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" event={"ID":"36effd34d282541d8cc7db737926b500","Type":"ContainerDied","Data":"cd671c3474087a46cf1cd4397fe83dcb408ac6d0227be0d35263fb0cf893b4c6"} Nov 03 22:07:12.923150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.923121 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" event={"ID":"36effd34d282541d8cc7db737926b500","Type":"ContainerStarted","Data":"a614e692f58d8f7e0cf3ec28a1e79ac57d2e7bcb798b87460a03fbfc01c9ca56"} Nov 03 22:07:12.923292 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.923246 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal" event={"ID":"a3ff6dbfd235d6871a32270ec8f4d659","Type":"ContainerStarted","Data":"78578b8cf38372b5ee7fe3f069d91600f6d27b8436d0645ef341bcc6b9fb7c3c"} Nov 03 22:07:12.923292 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.923263 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal" event={"ID":"a3ff6dbfd235d6871a32270ec8f4d659","Type":"ContainerStarted","Data":"a8218e7160e06cba3868bb17fb9f2219dbb98163d029a279ee76a91ec027ee86"} Nov 03 22:07:12.923292 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.923269 5084 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:12.961994 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.961963 5084 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:12.962179 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.962053 5084 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:12.986913 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.986883 5084 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:07:12.987055 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:12.986933 5084 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:12.987235 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:12.987220 5084 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:07:12.987273 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:12.987254 5084 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:13.056983 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.056938 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36effd34d282541d8cc7db737926b500-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal\" (UID: \"36effd34d282541d8cc7db737926b500\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:13.057161 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.056990 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36effd34d282541d8cc7db737926b500-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal\" (UID: \"36effd34d282541d8cc7db737926b500\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:13.057161 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.057009 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3ff6dbfd235d6871a32270ec8f4d659-config\") pod \"kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal\" (UID: \"a3ff6dbfd235d6871a32270ec8f4d659\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:13.158015 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.157925 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36effd34d282541d8cc7db737926b500-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal\" (UID: \"36effd34d282541d8cc7db737926b500\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:13.158015 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.157968 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36effd34d282541d8cc7db737926b500-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal\" (UID: \"36effd34d282541d8cc7db737926b500\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:13.158015 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.157987 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3ff6dbfd235d6871a32270ec8f4d659-config\") pod \"kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal\" (UID: \"a3ff6dbfd235d6871a32270ec8f4d659\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:13.158235 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.158036 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3ff6dbfd235d6871a32270ec8f4d659-config\") pod \"kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal\" (UID: \"a3ff6dbfd235d6871a32270ec8f4d659\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:13.158235 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.158042 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/36effd34d282541d8cc7db737926b500-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal\" (UID: \"36effd34d282541d8cc7db737926b500\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:13.158235 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.158042 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36effd34d282541d8cc7db737926b500-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal\" (UID: \"36effd34d282541d8cc7db737926b500\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:13.489898 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.489818 5084 apiserver.go:52] "Watching apiserver" Nov 03 22:07:13.600906 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.600867 5084 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:07:13.601648 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.601605 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rntwm","kube-system/global-pull-secret-syncer-8wmmt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6","openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf","openshift-multus/network-metrics-daemon-sfqbq","openshift-network-operator/iptables-alerter-gsmpr","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z","openshift-ovn-kubernetes/ovnkube-node-78tk2","openshift-insights/insights-runtime-extractor-vsnwk","openshift-ingress-canary/ingress-canary-w55xl","openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb","kube-system/kas-connection-checker-685895c7c9-tc9n2","openshift-multus/multus-additional-cni-plugins-2g9vv","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6nsjz","openshift-image-registry/node-ca-t6b4b","kube-system/kas-connection-checker-685895c7c9-z2q8s","openshift-ingress/router-default-84d846b7f5-4qjqn","openshift-network-console/networking-console-plugin-86c478bb88-p6mft","openshift-kube-storage-version-migrator/migrator-66df5d496b-5nc9k","openshift-kube-storage-version-migrator/migrator-66df5d496b-gpqdr","openshift-multus/multus-jl87t","openshift-dns/node-resolver-79sq5","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx","openshift-image-registry/image-registry-58d66d58db-phhsh","openshift-console-operator/console-operator-7cd56b8dbc-9mgrk","openshift-cluster-node-tuning-operator/tuned-pk5rg","openshift-service-ca/service-ca-5fffc97bdf-n4vt2","kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal","openshift-insights/periodic-gathering-wtfch-phz6j","kube-system/kas-connection-checker-685895c7c9-f88sj","kube-system/konnectivity-agent-k4rnh","openshift-insights/insights-operator-7c4986d449-fqp5x","openshift-image-registry/image-registry-595c4958c9-2rswm","openshift-network-diagnostics/network-check-source-6f78d59f99-q4b9r","openshift-network-diagnostics/network-check-target-z4ct8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal"] Nov 03 22:07:13.603482 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.603461 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:13.604376 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.604358 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:13.605244 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.605227 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.605718 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.605685 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:13.609458 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.609422 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:13.615304 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.615276 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:07:13.615766 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.615746 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:13.616874 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.616856 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:13.618153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.618136 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:13.651124 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.651091 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t6b4b_95246059-e5c1-4b42-9678-a1c8b2f63e80/node-ca/0.log" Nov 03 22:07:13.656677 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.656651 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 22:07:13.656812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.656698 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 22:07:13.657167 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.657152 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 22:07:13.657738 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.657711 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.657738 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.657711 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.657738 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.657736 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.657955 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.657936 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.657955 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.657949 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 22:07:13.658044 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.657958 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 22:07:13.658044 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.657966 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.658044 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.657991 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.658226 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.658206 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 22:07:13.658340 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.658254 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 22:07:13.658340 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.658311 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 22:07:13.658537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.658522 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 22:07:13.659049 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.659033 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 22:07:13.659146 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.659053 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.659146 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.659082 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 22:07:13.659146 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.659103 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.659146 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.659105 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 22:07:13.659338 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.659201 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.659338 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.659234 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.659338 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.659246 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.659338 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.659254 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.659338 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.659287 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.659338 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.659294 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.659338 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.659299 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 22:07:13.659526 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.659299 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 22:07:13.659833 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.659813 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.660043 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660017 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-etc-openvswitch\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.660137 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660036 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 22:07:13.660137 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660127 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-run-ovn\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.660237 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660172 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-node-log\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.660280 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660255 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-cni-bin\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.660311 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660292 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c0ac5b4-308e-4eca-8301-2c470cecdbef-ca-trust-extracted\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.660349 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660328 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-bound-sa-token\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.660403 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660382 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-var-lib-cni-multus\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.660473 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660425 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-run-multus-certs\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.660473 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660450 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c0ac5b4-308e-4eca-8301-2c470cecdbef-ca-trust-extracted\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.660473 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660461 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-run-systemd\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.660600 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660497 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-run-openvswitch\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.660600 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660538 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d49e0d2-566e-472c-9c2d-c85256d52de4-ovn-node-metrics-cert\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.660697 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660619 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f4d66c00-b01b-4c14-998a-d84707b4600b-iptables-alerter-script\") pod \"iptables-alerter-gsmpr\" (UID: \"f4d66c00-b01b-4c14-998a-d84707b4600b\") " pod="openshift-network-operator/iptables-alerter-gsmpr" Nov 03 22:07:13.660697 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660654 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3e7a098-088b-4e25-ac35-dc531d619b79-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:07:13.660792 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660718 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b3e7a098-088b-4e25-ac35-dc531d619b79-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:07:13.660792 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660763 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.660870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660836 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4d66c00-b01b-4c14-998a-d84707b4600b-host-slash\") pod \"iptables-alerter-gsmpr\" (UID: \"f4d66c00-b01b-4c14-998a-d84707b4600b\") " pod="openshift-network-operator/iptables-alerter-gsmpr" Nov 03 22:07:13.660941 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660898 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-hostroot\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.660941 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660932 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p6mft\" (UID: \"0a4cedce-72c5-4357-8001-23a5e14d7d9d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:13.661092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660965 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9kvp\" (UniqueName: \"kubernetes.io/projected/e099b6bc-addb-428b-9a88-2c8b8e544f46-kube-api-access-b9kvp\") pod \"service-ca-operator-544d749c8c-96jwb\" (UID: \"e099b6bc-addb-428b-9a88-2c8b8e544f46\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" Nov 03 22:07:13.661092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.660995 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3e7a098-088b-4e25-ac35-dc531d619b79-cnibin\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:07:13.661092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661021 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866a1e7a-4728-4bd5-a88e-1a97f36f1eac-tmp-dir\") pod \"node-resolver-79sq5\" (UID: \"866a1e7a-4728-4bd5-a88e-1a97f36f1eac\") " pod="openshift-dns/node-resolver-79sq5" Nov 03 22:07:13.661092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661049 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-cni-binary-copy\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.661271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661097 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-run-k8s-cni-cncf-io\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.661271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661128 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-var-lib-cni-bin\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.661271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661162 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn75q\" (UniqueName: \"kubernetes.io/projected/f4d66c00-b01b-4c14-998a-d84707b4600b-kube-api-access-fn75q\") pod \"iptables-alerter-gsmpr\" (UID: \"f4d66c00-b01b-4c14-998a-d84707b4600b\") " pod="openshift-network-operator/iptables-alerter-gsmpr" Nov 03 22:07:13.661271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661195 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e099b6bc-addb-428b-9a88-2c8b8e544f46-config\") pod \"service-ca-operator-544d749c8c-96jwb\" (UID: \"e099b6bc-addb-428b-9a88-2c8b8e544f46\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" Nov 03 22:07:13.661271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661224 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmppm\" (UniqueName: \"kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm\") pod \"network-check-target-z4ct8\" (UID: \"fab1061b-bf17-477e-bfdf-cb8b1d76b0ff\") " pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:07:13.661271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661254 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-conf-dir\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.661529 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661287 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b3e7a098-088b-4e25-ac35-dc531d619b79-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:07:13.661529 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661324 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdh26\" (UniqueName: \"kubernetes.io/projected/ec658817-ab95-4b41-a457-d067b9754e3e-kube-api-access-wdh26\") pod \"kas-connection-checker-685895c7c9-tc9n2\" (UID: \"ec658817-ab95-4b41-a457-d067b9754e3e\") " pod="kube-system/kas-connection-checker-685895c7c9-tc9n2" Nov 03 22:07:13.661529 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661348 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95246059-e5c1-4b42-9678-a1c8b2f63e80-host\") pod \"node-ca-t6b4b\" (UID: \"95246059-e5c1-4b42-9678-a1c8b2f63e80\") " pod="openshift-image-registry/node-ca-t6b4b" Nov 03 22:07:13.661529 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661381 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvmk9\" (UniqueName: \"kubernetes.io/projected/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-kube-api-access-tvmk9\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.661529 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661412 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1498363-2c98-4046-8feb-abb772a80149-serving-cert\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:13.661529 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661443 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1498363-2c98-4046-8feb-abb772a80149-trusted-ca\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:13.661529 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661473 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.661529 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661503 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-installation-pull-secrets\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.661862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661534 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-cni-dir\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.661862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661566 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whwxk\" (UniqueName: \"kubernetes.io/projected/ab2929c7-ab47-4660-80eb-f973bf6efdae-kube-api-access-whwxk\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:13.661862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661596 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-systemd-units\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.661862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661626 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d49e0d2-566e-472c-9c2d-c85256d52de4-ovnkube-script-lib\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.661862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661652 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e01c32-cde7-4c5e-add4-ca8c17e82e34-tmp\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:13.661862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661682 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgntw\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-kube-api-access-qgntw\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.661862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661722 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e099b6bc-addb-428b-9a88-2c8b8e544f46-serving-cert\") pod \"service-ca-operator-544d749c8c-96jwb\" (UID: \"e099b6bc-addb-428b-9a88-2c8b8e544f46\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" Nov 03 22:07:13.661862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661761 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:13.661862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661787 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95246059-e5c1-4b42-9678-a1c8b2f63e80-serviceca\") pod \"node-ca-t6b4b\" (UID: \"95246059-e5c1-4b42-9678-a1c8b2f63e80\") " pod="openshift-image-registry/node-ca-t6b4b" Nov 03 22:07:13.661862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661815 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-log-socket\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.661862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661849 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-cni-netd\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.662361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661879 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d49e0d2-566e-472c-9c2d-c85256d52de4-env-overrides\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.662361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661910 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60e01c32-cde7-4c5e-add4-ca8c17e82e34-serving-cert\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:13.662361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661940 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgxtv\" (UniqueName: \"kubernetes.io/projected/95246059-e5c1-4b42-9678-a1c8b2f63e80-kube-api-access-lgxtv\") pod \"node-ca-t6b4b\" (UID: \"95246059-e5c1-4b42-9678-a1c8b2f63e80\") " pod="openshift-image-registry/node-ca-t6b4b" Nov 03 22:07:13.662361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661970 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-run-ovn-kubernetes\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.662361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.661999 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vt2f\" (UniqueName: \"kubernetes.io/projected/1d49e0d2-566e-472c-9c2d-c85256d52de4-kube-api-access-6vt2f\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.662361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662031 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/866a1e7a-4728-4bd5-a88e-1a97f36f1eac-hosts-file\") pod \"node-resolver-79sq5\" (UID: \"866a1e7a-4728-4bd5-a88e-1a97f36f1eac\") " pod="openshift-dns/node-resolver-79sq5" Nov 03 22:07:13.662361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662078 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:13.662361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662111 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grqtw\" (UniqueName: \"kubernetes.io/projected/2e0d6adb-5887-42af-ad83-48863e1ac386-kube-api-access-grqtw\") pod \"network-check-source-6f78d59f99-q4b9r\" (UID: \"2e0d6adb-5887-42af-ad83-48863e1ac386\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q4b9r" Nov 03 22:07:13.662361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662148 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-slash\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.662361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662175 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z\" (UID: \"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" Nov 03 22:07:13.662361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662208 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-default-certificate\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:13.662361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662240 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzd9z\" (UniqueName: \"kubernetes.io/projected/508293cb-d657-4cfb-a67a-0155a44d6e47-kube-api-access-nzd9z\") pod \"kas-connection-checker-685895c7c9-z2q8s\" (UID: \"508293cb-d657-4cfb-a67a-0155a44d6e47\") " pod="kube-system/kas-connection-checker-685895c7c9-z2q8s" Nov 03 22:07:13.662361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662274 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e01c32-cde7-4c5e-add4-ca8c17e82e34-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:13.662361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662302 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e01c32-cde7-4c5e-add4-ca8c17e82e34-service-ca-bundle\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:13.662361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662335 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v728z\" (UniqueName: \"kubernetes.io/projected/43929f2c-6b7a-4f25-8a97-51f6bee5481a-kube-api-access-v728z\") pod \"volume-data-source-validator-58c9f5f46d-6nsjz\" (UID: \"43929f2c-6b7a-4f25-8a97-51f6bee5481a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6nsjz" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662369 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-image-registry-private-configuration\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662403 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm7gx\" (UniqueName: \"kubernetes.io/projected/866a1e7a-4728-4bd5-a88e-1a97f36f1eac-kube-api-access-sm7gx\") pod \"node-resolver-79sq5\" (UID: \"866a1e7a-4728-4bd5-a88e-1a97f36f1eac\") " pod="openshift-dns/node-resolver-79sq5" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662436 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zt5p\" (UniqueName: \"kubernetes.io/projected/f2a09669-a5d7-4dbb-9ddf-842f4f919028-kube-api-access-8zt5p\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662463 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-cnibin\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662494 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-socket-dir-parent\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662526 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-var-lib-kubelet\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662558 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3e7a098-088b-4e25-ac35-dc531d619b79-cni-binary-copy\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662592 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-run-netns\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662618 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-var-lib-openvswitch\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662673 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d49e0d2-566e-472c-9c2d-c85256d52de4-ovnkube-config\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662707 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx9rl\" (UniqueName: \"kubernetes.io/projected/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3-kube-api-access-fx9rl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z\" (UID: \"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662739 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-trusted-ca\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662766 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3e7a098-088b-4e25-ac35-dc531d619b79-system-cni-dir\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662800 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlwbf\" (UniqueName: \"kubernetes.io/projected/b3e7a098-088b-4e25-ac35-dc531d619b79-kube-api-access-vlwbf\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662833 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-kubelet\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662864 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-system-cni-dir\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662894 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-run-netns\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.662957 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-certificates\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.663008 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z\" (UID: \"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" Nov 03 22:07:13.663054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.663040 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-stats-auth\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:13.664037 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.663086 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:13.664037 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.663123 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjj86\" (UniqueName: \"kubernetes.io/projected/d1498363-2c98-4046-8feb-abb772a80149-kube-api-access-tjj86\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:13.664037 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.663167 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/60e01c32-cde7-4c5e-add4-ca8c17e82e34-snapshots\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:13.664037 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.663203 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz8gb\" (UniqueName: \"kubernetes.io/projected/60e01c32-cde7-4c5e-add4-ca8c17e82e34-kube-api-access-kz8gb\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:13.664037 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.663241 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:13.664037 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.663274 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptsfs\" (UniqueName: \"kubernetes.io/projected/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-kube-api-access-ptsfs\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:13.664037 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.663302 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfhnj\" (UniqueName: \"kubernetes.io/projected/fe0af137-f9ce-4f18-8329-3f7df6a728bb-kube-api-access-bfhnj\") pod \"kas-connection-checker-685895c7c9-f88sj\" (UID: \"fe0af137-f9ce-4f18-8329-3f7df6a728bb\") " pod="kube-system/kas-connection-checker-685895c7c9-f88sj" Nov 03 22:07:13.664037 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.663337 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0a4cedce-72c5-4357-8001-23a5e14d7d9d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-p6mft\" (UID: \"0a4cedce-72c5-4357-8001-23a5e14d7d9d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:13.664037 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.663372 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:13.664037 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.663405 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d1498363-2c98-4046-8feb-abb772a80149-config\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:13.664037 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.663435 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-daemon-config\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.664037 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.663469 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-etc-kubernetes\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.664037 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.663502 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ab2929c7-ab47-4660-80eb-f973bf6efdae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:13.664037 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.663776 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/60e01c32-cde7-4c5e-add4-ca8c17e82e34-snapshots\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:13.664037 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.663884 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ab2929c7-ab47-4660-80eb-f973bf6efdae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:13.664037 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.663898 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866a1e7a-4728-4bd5-a88e-1a97f36f1eac-tmp-dir\") pod \"node-resolver-79sq5\" (UID: \"866a1e7a-4728-4bd5-a88e-1a97f36f1eac\") " pod="openshift-dns/node-resolver-79sq5" Nov 03 22:07:13.664819 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.664210 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e099b6bc-addb-428b-9a88-2c8b8e544f46-config\") pod \"service-ca-operator-544d749c8c-96jwb\" (UID: \"e099b6bc-addb-428b-9a88-2c8b8e544f46\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" Nov 03 22:07:13.664819 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.664384 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e099b6bc-addb-428b-9a88-2c8b8e544f46-serving-cert\") pod \"service-ca-operator-544d749c8c-96jwb\" (UID: \"e099b6bc-addb-428b-9a88-2c8b8e544f46\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" Nov 03 22:07:13.664819 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.664387 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-installation-pull-secrets\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.664819 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.664432 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e01c32-cde7-4c5e-add4-ca8c17e82e34-tmp\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:13.664819 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.664698 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60e01c32-cde7-4c5e-add4-ca8c17e82e34-serving-cert\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:13.664819 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.664792 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z\" (UID: \"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" Nov 03 22:07:13.665236 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.665213 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e01c32-cde7-4c5e-add4-ca8c17e82e34-service-ca-bundle\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:13.665236 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.665226 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-default-certificate\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:13.665375 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.665273 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z\" (UID: \"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" Nov 03 22:07:13.665416 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.665404 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-stats-auth\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:13.665936 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.665907 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-image-registry-private-configuration\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.666104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.666081 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e01c32-cde7-4c5e-add4-ca8c17e82e34-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:13.701364 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.701334 5084 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:13.701523 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.701465 5084 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:13.702138 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.702120 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 22:07:13.702463 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.702446 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 22:07:13.702972 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.702961 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 22:07:13.703520 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.703505 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.703593 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.703521 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 22:07:13.703593 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.703532 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.703593 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.703591 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.703682 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.703664 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.704666 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.704649 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d1498363-2c98-4046-8feb-abb772a80149-config\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:13.704864 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.704849 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d1498363-2c98-4046-8feb-abb772a80149-serving-cert\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:13.705755 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.705738 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0a4cedce-72c5-4357-8001-23a5e14d7d9d-nginx-conf\") pod \"networking-console-plugin-86c478bb88-p6mft\" (UID: \"0a4cedce-72c5-4357-8001-23a5e14d7d9d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:13.707907 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.707892 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.710518 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710488 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 22:07:13.710518 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710492 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 22:07:13.710707 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710535 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:07:13.710707 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710543 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.710707 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710627 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.710707 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710632 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.710707 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710636 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710710 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710732 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710739 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710746 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710739 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710761 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710788 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710792 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710804 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710817 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710828 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710853 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710861 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710828 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710874 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710872 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710887 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710907 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710914 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710932 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710941 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710935 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.710970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.710964 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711001 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711011 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b3e7a098-088b-4e25-ac35-dc531d619b79-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711028 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711056 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711034 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711038 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711097 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dgskz\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711111 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8pftz\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711117 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711127 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ckrtj\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711035 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bkplg\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711130 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5mdhj\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711154 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711162 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711175 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ng2g2\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711202 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kxpq\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711227 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711288 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z5g9w\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711344 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-28d69\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711381 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711419 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711495 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2d6mp\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711499 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-g6ff6\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711571 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gsv4s\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711600 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711642 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m27qh\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711711 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fcb7n\"" Nov 03 22:07:13.711780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711760 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 22:07:13.712657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711821 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dngqk\"" Nov 03 22:07:13.712657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711856 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:07:13.712657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711985 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 22:07:13.712657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.711985 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbwbg\"" Nov 03 22:07:13.712657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.712025 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-w8n9w\"" Nov 03 22:07:13.712657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.712123 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 22:07:13.712657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.712168 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cqv8d\"" Nov 03 22:07:13.712657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.712228 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j54w4\"" Nov 03 22:07:13.712657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.712305 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:07:13.713130 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.713115 5084 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 22:07:13.714357 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.714341 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-cni-binary-copy\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.714404 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.714366 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1d49e0d2-566e-472c-9c2d-c85256d52de4-ovnkube-script-lib\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.714442 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.714405 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1d49e0d2-566e-472c-9c2d-c85256d52de4-ovnkube-config\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.714442 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.714412 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-daemon-config\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.715480 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.715457 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1d49e0d2-566e-472c-9c2d-c85256d52de4-env-overrides\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.715610 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.715485 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b3e7a098-088b-4e25-ac35-dc531d619b79-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:07:13.715610 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.715575 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-certificates\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.715610 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.715598 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95246059-e5c1-4b42-9678-a1c8b2f63e80-serviceca\") pod \"node-ca-t6b4b\" (UID: \"95246059-e5c1-4b42-9678-a1c8b2f63e80\") " pod="openshift-image-registry/node-ca-t6b4b" Nov 03 22:07:13.715738 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.715613 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1498363-2c98-4046-8feb-abb772a80149-trusted-ca\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:13.715738 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.715643 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:13.715738 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.715649 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-trusted-ca\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.715738 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.715708 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3e7a098-088b-4e25-ac35-dc531d619b79-cni-binary-copy\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:07:13.716231 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.716215 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-bound-sa-token\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.721271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.720985 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d49e0d2-566e-472c-9c2d-c85256d52de4-ovn-node-metrics-cert\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.721271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.721130 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f4d66c00-b01b-4c14-998a-d84707b4600b-iptables-alerter-script\") pod \"iptables-alerter-gsmpr\" (UID: \"f4d66c00-b01b-4c14-998a-d84707b4600b\") " pod="openshift-network-operator/iptables-alerter-gsmpr" Nov 03 22:07:13.721271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.721156 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx9rl\" (UniqueName: \"kubernetes.io/projected/e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3-kube-api-access-fx9rl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z\" (UID: \"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" Nov 03 22:07:13.721271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.721133 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgxtv\" (UniqueName: \"kubernetes.io/projected/95246059-e5c1-4b42-9678-a1c8b2f63e80-kube-api-access-lgxtv\") pod \"node-ca-t6b4b\" (UID: \"95246059-e5c1-4b42-9678-a1c8b2f63e80\") " pod="openshift-image-registry/node-ca-t6b4b" Nov 03 22:07:13.721271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.721154 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfhnj\" (UniqueName: \"kubernetes.io/projected/fe0af137-f9ce-4f18-8329-3f7df6a728bb-kube-api-access-bfhnj\") pod \"kas-connection-checker-685895c7c9-f88sj\" (UID: \"fe0af137-f9ce-4f18-8329-3f7df6a728bb\") " pod="kube-system/kas-connection-checker-685895c7c9-f88sj" Nov 03 22:07:13.721271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.721193 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm7gx\" (UniqueName: \"kubernetes.io/projected/866a1e7a-4728-4bd5-a88e-1a97f36f1eac-kube-api-access-sm7gx\") pod \"node-resolver-79sq5\" (UID: \"866a1e7a-4728-4bd5-a88e-1a97f36f1eac\") " pod="openshift-dns/node-resolver-79sq5" Nov 03 22:07:13.721667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.721298 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzd9z\" (UniqueName: \"kubernetes.io/projected/508293cb-d657-4cfb-a67a-0155a44d6e47-kube-api-access-nzd9z\") pod \"kas-connection-checker-685895c7c9-z2q8s\" (UID: \"508293cb-d657-4cfb-a67a-0155a44d6e47\") " pod="kube-system/kas-connection-checker-685895c7c9-z2q8s" Nov 03 22:07:13.721667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.721426 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvmk9\" (UniqueName: \"kubernetes.io/projected/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-kube-api-access-tvmk9\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.721667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.721560 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vt2f\" (UniqueName: \"kubernetes.io/projected/1d49e0d2-566e-472c-9c2d-c85256d52de4-kube-api-access-6vt2f\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.721667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.721626 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn75q\" (UniqueName: \"kubernetes.io/projected/f4d66c00-b01b-4c14-998a-d84707b4600b-kube-api-access-fn75q\") pod \"iptables-alerter-gsmpr\" (UID: \"f4d66c00-b01b-4c14-998a-d84707b4600b\") " pod="openshift-network-operator/iptables-alerter-gsmpr" Nov 03 22:07:13.721883 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.721870 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlwbf\" (UniqueName: \"kubernetes.io/projected/b3e7a098-088b-4e25-ac35-dc531d619b79-kube-api-access-vlwbf\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:07:13.722054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.722033 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmppm\" (UniqueName: \"kubernetes.io/projected/fab1061b-bf17-477e-bfdf-cb8b1d76b0ff-kube-api-access-jmppm\") pod \"network-check-target-z4ct8\" (UID: \"fab1061b-bf17-477e-bfdf-cb8b1d76b0ff\") " pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:07:13.722161 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.722146 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptsfs\" (UniqueName: \"kubernetes.io/projected/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-kube-api-access-ptsfs\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:13.757739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.757135 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz8gb\" (UniqueName: \"kubernetes.io/projected/60e01c32-cde7-4c5e-add4-ca8c17e82e34-kube-api-access-kz8gb\") pod \"insights-operator-7c4986d449-fqp5x\" (UID: \"60e01c32-cde7-4c5e-add4-ca8c17e82e34\") " pod="openshift-insights/insights-operator-7c4986d449-fqp5x" Nov 03 22:07:13.757739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.757611 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgntw\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-kube-api-access-qgntw\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.757739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.757628 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9kvp\" (UniqueName: \"kubernetes.io/projected/e099b6bc-addb-428b-9a88-2c8b8e544f46-kube-api-access-b9kvp\") pod \"service-ca-operator-544d749c8c-96jwb\" (UID: \"e099b6bc-addb-428b-9a88-2c8b8e544f46\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" Nov 03 22:07:13.757739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.757647 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdh26\" (UniqueName: \"kubernetes.io/projected/ec658817-ab95-4b41-a457-d067b9754e3e-kube-api-access-wdh26\") pod \"kas-connection-checker-685895c7c9-tc9n2\" (UID: \"ec658817-ab95-4b41-a457-d067b9754e3e\") " pod="kube-system/kas-connection-checker-685895c7c9-tc9n2" Nov 03 22:07:13.757739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.757650 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zt5p\" (UniqueName: \"kubernetes.io/projected/f2a09669-a5d7-4dbb-9ddf-842f4f919028-kube-api-access-8zt5p\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:13.757739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.757686 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whwxk\" (UniqueName: \"kubernetes.io/projected/ab2929c7-ab47-4660-80eb-f973bf6efdae-kube-api-access-whwxk\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:13.758014 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.757776 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v728z\" (UniqueName: \"kubernetes.io/projected/43929f2c-6b7a-4f25-8a97-51f6bee5481a-kube-api-access-v728z\") pod \"volume-data-source-validator-58c9f5f46d-6nsjz\" (UID: \"43929f2c-6b7a-4f25-8a97-51f6bee5481a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-6nsjz" Nov 03 22:07:13.758014 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.757858 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjj86\" (UniqueName: \"kubernetes.io/projected/d1498363-2c98-4046-8feb-abb772a80149-kube-api-access-tjj86\") pod \"console-operator-7cd56b8dbc-9mgrk\" (UID: \"d1498363-2c98-4046-8feb-abb772a80149\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:13.758014 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.757956 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grqtw\" (UniqueName: \"kubernetes.io/projected/2e0d6adb-5887-42af-ad83-48863e1ac386-kube-api-access-grqtw\") pod \"network-check-source-6f78d59f99-q4b9r\" (UID: \"2e0d6adb-5887-42af-ad83-48863e1ac386\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q4b9r" Nov 03 22:07:13.764027 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.763996 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aacf527d-b30b-48cd-b5fd-be3eba9a2095-ca-trust-extracted\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:13.764163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764033 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-kubernetes\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.764163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764052 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.764163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764088 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4d66c00-b01b-4c14-998a-d84707b4600b-host-slash\") pod \"iptables-alerter-gsmpr\" (UID: \"f4d66c00-b01b-4c14-998a-d84707b4600b\") " pod="openshift-network-operator/iptables-alerter-gsmpr" Nov 03 22:07:13.764163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764104 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-trusted-ca\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:13.764362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764158 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-hostroot\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.764362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764173 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f4d66c00-b01b-4c14-998a-d84707b4600b-host-slash\") pod \"iptables-alerter-gsmpr\" (UID: \"f4d66c00-b01b-4c14-998a-d84707b4600b\") " pod="openshift-network-operator/iptables-alerter-gsmpr" Nov 03 22:07:13.764362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764218 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p6mft\" (UID: \"0a4cedce-72c5-4357-8001-23a5e14d7d9d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:13.764362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764250 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-hostroot\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.764362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764254 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-image-registry-private-configuration\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:13.764362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764261 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aacf527d-b30b-48cd-b5fd-be3eba9a2095-ca-trust-extracted\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:13.764362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764291 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf54n\" (UniqueName: \"kubernetes.io/projected/d3b08229-d363-4336-bb37-89df96b43dbe-kube-api-access-gf54n\") pod \"ingress-canary-w55xl\" (UID: \"d3b08229-d363-4336-bb37-89df96b43dbe\") " pod="openshift-ingress-canary/ingress-canary-w55xl" Nov 03 22:07:13.764362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764312 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-conf-dir\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.764362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764341 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95246059-e5c1-4b42-9678-a1c8b2f63e80-host\") pod \"node-ca-t6b4b\" (UID: \"95246059-e5c1-4b42-9678-a1c8b2f63e80\") " pod="openshift-image-registry/node-ca-t6b4b" Nov 03 22:07:13.764362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764266 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.764362 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:13.764346 5084 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764370 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drgl\" (UniqueName: \"kubernetes.io/projected/412af756-be10-401f-8413-419d66bafaad-kube-api-access-4drgl\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764398 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzpgz\" (UniqueName: \"kubernetes.io/projected/19c22c7c-c7e9-4ef8-9e55-245393662f36-kube-api-access-wzpgz\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764402 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-conf-dir\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764428 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95246059-e5c1-4b42-9678-a1c8b2f63e80-host\") pod \"node-ca-t6b4b\" (UID: \"95246059-e5c1-4b42-9678-a1c8b2f63e80\") " pod="openshift-image-registry/node-ca-t6b4b" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:13.764436 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert podName:0a4cedce-72c5-4357-8001-23a5e14d7d9d nodeName:}" failed. No retries permitted until 2026-11-03 22:07:14.264412816 +0000 UTC m=+1.858444789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p6mft" (UID: "0a4cedce-72c5-4357-8001-23a5e14d7d9d") : secret "networking-console-plugin-cert" not found Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764462 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-image-registry-private-configuration\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764492 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764527 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-modprobe-d\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764547 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-trusted-ca\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764570 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-certificates\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764604 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-log-socket\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764628 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-cni-netd\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764654 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdhtw\" (UniqueName: \"kubernetes.io/projected/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad-kube-api-access-gdhtw\") pod \"service-ca-5fffc97bdf-n4vt2\" (UID: \"46cb82c6-4e9d-49cd-8bc6-0e006d3785ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764668 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-log-socket\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764683 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-run-ovn-kubernetes\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764707 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-sysctl-d\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764725 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-run-ovn-kubernetes\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764731 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764740 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-certificates\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764743 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-cni-netd\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764779 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-trusted-ca-bundle\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764821 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/866a1e7a-4728-4bd5-a88e-1a97f36f1eac-hosts-file\") pod \"node-resolver-79sq5\" (UID: \"866a1e7a-4728-4bd5-a88e-1a97f36f1eac\") " pod="openshift-dns/node-resolver-79sq5" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764848 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:13.764866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764877 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764896 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/866a1e7a-4728-4bd5-a88e-1a97f36f1eac-hosts-file\") pod \"node-resolver-79sq5\" (UID: \"866a1e7a-4728-4bd5-a88e-1a97f36f1eac\") " pod="openshift-dns/node-resolver-79sq5" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764899 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-socket-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.764946 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-registration-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:13.764950 5084 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765000 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/19c22c7c-c7e9-4ef8-9e55-245393662f36-archives-path\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:13.765031 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls podName:ab2929c7-ab47-4660-80eb-f973bf6efdae nodeName:}" failed. No retries permitted until 2026-11-03 22:07:14.265012093 +0000 UTC m=+1.859044052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8gtxf" (UID: "ab2929c7-ab47-4660-80eb-f973bf6efdae") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765079 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-trusted-ca-bundle\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765092 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-var-lib-openvswitch\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765097 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/19c22c7c-c7e9-4ef8-9e55-245393662f36-archives-path\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765111 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-socket-dir-parent\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765127 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-var-lib-kubelet\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765132 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-var-lib-openvswitch\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765147 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-sys-fs\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765154 5084 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765162 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-var-lib-kubelet\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765171 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-n4vt2\" (UID: \"46cb82c6-4e9d-49cd-8bc6-0e006d3785ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765190 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-socket-dir-parent\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765200 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-systemd\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:13.765208 5084 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765252 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27hj2\" (UniqueName: \"kubernetes.io/projected/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-api-access-27hj2\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765333 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-n4vt2\" (UID: \"46cb82c6-4e9d-49cd-8bc6-0e006d3785ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765338 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-service-ca-bundle\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765363 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-kubelet\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765380 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19a326ff-4502-4590-9c0d-194f8cf14abd-konnectivity-ca\") pod \"konnectivity-agent-k4rnh\" (UID: \"19a326ff-4502-4590-9c0d-194f8cf14abd\") " pod="kube-system/konnectivity-agent-k4rnh" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765402 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-device-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765417 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-kubelet\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765422 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765432 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-service-ca-bundle\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765540 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-run-multus-certs\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765556 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-run-multus-certs\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765629 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99d8e137-a970-441d-9800-d1ce1c8f1f94-dbus\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765679 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-sysconfig\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765701 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/19a326ff-4502-4590-9c0d-194f8cf14abd-konnectivity-ca\") pod \"konnectivity-agent-k4rnh\" (UID: \"19a326ff-4502-4590-9c0d-194f8cf14abd\") " pod="kube-system/konnectivity-agent-k4rnh" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765726 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765760 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765786 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-cni-bin\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765813 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-lib-modules\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:13.765828 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle podName:b537a9eb-5abf-4f6b-9a35-0f13c5211bf2 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:14.265810701 +0000 UTC m=+1.859842690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle") pod "router-default-84d846b7f5-4qjqn" (UID: "b537a9eb-5abf-4f6b-9a35-0f13c5211bf2") : configmap references non-existent config key: service-ca.crt Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765853 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-cni-bin\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765861 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-host\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:13.765878 5084 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765892 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765903 5084 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:13.765915 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls podName:f2a09669-a5d7-4dbb-9ddf-842f4f919028 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:14.265904369 +0000 UTC m=+1.859936324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dfffx" (UID: "f2a09669-a5d7-4dbb-9ddf-842f4f919028") : secret "samples-operator-tls" not found Nov 03 22:07:13.765924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765937 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-run-ovn\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765955 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-etc-selinux\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.765972 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99d8e137-a970-441d-9800-d1ce1c8f1f94-kubelet-config\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766026 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-run-ovn\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766080 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766107 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-var-lib-cni-multus\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766123 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-run-openvswitch\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766140 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8cv4\" (UniqueName: \"kubernetes.io/projected/43b5a34d-6354-4586-9dd1-55279e0dcfad-kube-api-access-m8cv4\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766156 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q9b5\" (UniqueName: \"kubernetes.io/projected/fdc0640a-6da9-4cd2-a6a8-07d56a91718e-kube-api-access-9q9b5\") pod \"migrator-66df5d496b-gpqdr\" (UID: \"fdc0640a-6da9-4cd2-a6a8-07d56a91718e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpqdr" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766165 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-var-lib-cni-multus\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766177 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3e7a098-088b-4e25-ac35-dc531d619b79-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766180 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-run-openvswitch\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766198 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c059f8a-ed76-4ffb-9f24-be79941af384-data-volume\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766222 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766248 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3e7a098-088b-4e25-ac35-dc531d619b79-cnibin\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766251 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3e7a098-088b-4e25-ac35-dc531d619b79-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766272 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad-signing-key\") pod \"service-ca-5fffc97bdf-n4vt2\" (UID: \"46cb82c6-4e9d-49cd-8bc6-0e006d3785ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766296 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c059f8a-ed76-4ffb-9f24-be79941af384-data-volume\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766315 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-run-k8s-cni-cncf-io\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766329 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3e7a098-088b-4e25-ac35-dc531d619b79-cnibin\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766340 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-var-lib-cni-bin\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766380 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-var-lib-cni-bin\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766387 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-installation-pull-secrets\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766399 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-run-k8s-cni-cncf-io\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766422 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-bound-sa-token\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766451 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms2cm\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-kube-api-access-ms2cm\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766458 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad-signing-key\") pod \"service-ca-5fffc97bdf-n4vt2\" (UID: \"46cb82c6-4e9d-49cd-8bc6-0e006d3785ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766474 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b599076-85db-4b98-8636-485e964a5e59-tmp\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766543 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b599076-85db-4b98-8636-485e964a5e59-tmp\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766573 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c059f8a-ed76-4ffb-9f24-be79941af384-crio-socket\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766593 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-run\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766593 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-installation-pull-secrets\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766615 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766660 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-cni-dir\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766717 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99d8e137-a970-441d-9800-d1ce1c8f1f94-original-pull-secret\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766752 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-multus-cni-dir\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766807 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-systemd-units\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766836 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766848 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-systemd-units\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766886 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-sys\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766943 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b5a34d-6354-4586-9dd1-55279e0dcfad-tmp-dir\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766968 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-slash\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.766986 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3b08229-d363-4336-bb37-89df96b43dbe-cert\") pod \"ingress-canary-w55xl\" (UID: \"d3b08229-d363-4336-bb37-89df96b43dbe\") " pod="openshift-ingress-canary/ingress-canary-w55xl" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767003 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz5dx\" (UniqueName: \"kubernetes.io/projected/8b599076-85db-4b98-8636-485e964a5e59-kube-api-access-sz5dx\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767029 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb6r6\" (UniqueName: \"kubernetes.io/projected/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-kube-api-access-vb6r6\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767038 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-slash\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767109 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-cnibin\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767132 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-run-netns\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767150 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43b5a34d-6354-4586-9dd1-55279e0dcfad-tmp-dir\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767162 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19a326ff-4502-4590-9c0d-194f8cf14abd-agent-certs\") pod \"konnectivity-agent-k4rnh\" (UID: \"19a326ff-4502-4590-9c0d-194f8cf14abd\") " pod="kube-system/konnectivity-agent-k4rnh" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767186 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-cnibin\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767193 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-var-lib-kubelet\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767223 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3e7a098-088b-4e25-ac35-dc531d619b79-system-cni-dir\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767235 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-host-run-netns\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767248 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43b5a34d-6354-4586-9dd1-55279e0dcfad-config-volume\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767274 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-sysctl-conf\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767311 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-system-cni-dir\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767337 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-run-netns\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767347 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3b08229-d363-4336-bb37-89df96b43dbe-cert\") pod \"ingress-canary-w55xl\" (UID: \"d3b08229-d363-4336-bb37-89df96b43dbe\") " pod="openshift-ingress-canary/ingress-canary-w55xl" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767384 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-run-systemd\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767390 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-host-run-netns\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767439 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-run-systemd\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767442 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6xt\" (UniqueName: \"kubernetes.io/projected/3da2c3b3-3a29-48e6-8281-8bdcd184b3f0-kube-api-access-fh6xt\") pod \"migrator-66df5d496b-5nc9k\" (UID: \"3da2c3b3-3a29-48e6-8281-8bdcd184b3f0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5nc9k" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767497 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43b5a34d-6354-4586-9dd1-55279e0dcfad-config-volume\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767498 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/19a326ff-4502-4590-9c0d-194f8cf14abd-agent-certs\") pod \"konnectivity-agent-k4rnh\" (UID: \"19a326ff-4502-4590-9c0d-194f8cf14abd\") " pod="kube-system/konnectivity-agent-k4rnh" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767515 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-system-cni-dir\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767538 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b599076-85db-4b98-8636-485e964a5e59-etc-tuned\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767511 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3e7a098-088b-4e25-ac35-dc531d619b79-system-cni-dir\") pod \"multus-additional-cni-plugins-2g9vv\" (UID: \"b3e7a098-088b-4e25-ac35-dc531d619b79\") " pod="openshift-multus/multus-additional-cni-plugins-2g9vv" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767637 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-etc-kubernetes\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767700 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b599076-85db-4b98-8636-485e964a5e59-etc-tuned\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767706 5084 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767715 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37b1fdc9-35d8-4bd3-af0c-cea9299192e8-etc-kubernetes\") pod \"multus-jl87t\" (UID: \"37b1fdc9-35d8-4bd3-af0c-cea9299192e8\") " pod="openshift-multus/multus-jl87t" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767700 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-etc-openvswitch\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:13.767755 5084 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.us-west-2.compute.internal" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767761 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-etc-openvswitch\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767746 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-node-log\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.768111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.767789 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1d49e0d2-566e-472c-9c2d-c85256d52de4-node-log\") pod \"ovnkube-node-78tk2\" (UID: \"1d49e0d2-566e-472c-9c2d-c85256d52de4\") " pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:13.771278 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.770289 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls\") pod \"image-registry-595c4958c9-2rswm\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.771278 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.770358 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-metrics-certs\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:13.821890 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.821843 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27hj2\" (UniqueName: \"kubernetes.io/projected/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-api-access-27hj2\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:13.822387 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.822356 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4drgl\" (UniqueName: \"kubernetes.io/projected/412af756-be10-401f-8413-419d66bafaad-kube-api-access-4drgl\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.822484 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.822389 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzpgz\" (UniqueName: \"kubernetes.io/projected/19c22c7c-c7e9-4ef8-9e55-245393662f36-kube-api-access-wzpgz\") pod \"periodic-gathering-wtfch-phz6j\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:13.822484 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.822452 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf54n\" (UniqueName: \"kubernetes.io/projected/d3b08229-d363-4336-bb37-89df96b43dbe-kube-api-access-gf54n\") pod \"ingress-canary-w55xl\" (UID: \"d3b08229-d363-4336-bb37-89df96b43dbe\") " pod="openshift-ingress-canary/ingress-canary-w55xl" Nov 03 22:07:13.822594 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.822514 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdhtw\" (UniqueName: \"kubernetes.io/projected/46cb82c6-4e9d-49cd-8bc6-0e006d3785ad-kube-api-access-gdhtw\") pod \"service-ca-5fffc97bdf-n4vt2\" (UID: \"46cb82c6-4e9d-49cd-8bc6-0e006d3785ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-n4vt2" Nov 03 22:07:13.823110 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.823091 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms2cm\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-kube-api-access-ms2cm\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:13.823347 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.823327 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8cv4\" (UniqueName: \"kubernetes.io/projected/43b5a34d-6354-4586-9dd1-55279e0dcfad-kube-api-access-m8cv4\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:13.823636 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.823620 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q9b5\" (UniqueName: \"kubernetes.io/projected/fdc0640a-6da9-4cd2-a6a8-07d56a91718e-kube-api-access-9q9b5\") pod \"migrator-66df5d496b-gpqdr\" (UID: \"fdc0640a-6da9-4cd2-a6a8-07d56a91718e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gpqdr" Nov 03 22:07:13.825666 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.825645 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6xt\" (UniqueName: \"kubernetes.io/projected/3da2c3b3-3a29-48e6-8281-8bdcd184b3f0-kube-api-access-fh6xt\") pod \"migrator-66df5d496b-5nc9k\" (UID: \"3da2c3b3-3a29-48e6-8281-8bdcd184b3f0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5nc9k" Nov 03 22:07:13.826003 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.825981 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz5dx\" (UniqueName: \"kubernetes.io/projected/8b599076-85db-4b98-8636-485e964a5e59-kube-api-access-sz5dx\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.826364 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.826347 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-bound-sa-token\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:13.826402 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.826372 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb6r6\" (UniqueName: \"kubernetes.io/projected/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-kube-api-access-vb6r6\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:07:13.868506 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868458 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c059f8a-ed76-4ffb-9f24-be79941af384-crio-socket\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:13.868506 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868500 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-run\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.868743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868521 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.868743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868583 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-run\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.868743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868595 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c059f8a-ed76-4ffb-9f24-be79941af384-crio-socket\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:13.868743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868636 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-sys\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.868743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868659 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.868743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868719 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-sys\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.868995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868750 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-var-lib-kubelet\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.868995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868766 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-sysctl-conf\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.868995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868789 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-kubernetes\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.868995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868821 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-var-lib-kubelet\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.868995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868835 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-kubernetes\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.868995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868866 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:13.868995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868883 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-modprobe-d\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.868995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868905 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-sysctl-d\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.868995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868923 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:07:13.868995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868933 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-sysctl-conf\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.868995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868949 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:13.868995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868973 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-socket-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.868995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.868991 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-registration-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869013 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-sys-fs\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869024 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-sysctl-d\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869032 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-systemd\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:13.869027 5084 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869034 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-modprobe-d\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:13.869046 5084 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869106 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-socket-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869107 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-device-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:13.869132 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls podName:43b5a34d-6354-4586-9dd1-55279e0dcfad nodeName:}" failed. No retries permitted until 2026-11-03 22:07:14.369111648 +0000 UTC m=+1.963143635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls") pod "dns-default-rntwm" (UID: "43b5a34d-6354-4586-9dd1-55279e0dcfad") : secret "dns-default-metrics-tls" not found Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869132 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-systemd\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869151 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-device-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869149 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-sys-fs\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:13.869170 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs podName:393f28cb-5fc8-4b3e-8852-eebe41b42a2b nodeName:}" failed. No retries permitted until 2026-11-03 22:07:14.369152351 +0000 UTC m=+1.963184304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs") pod "network-metrics-daemon-sfqbq" (UID: "393f28cb-5fc8-4b3e-8852-eebe41b42a2b") : secret "metrics-daemon-secret" not found Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869161 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-registration-dir\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869216 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99d8e137-a970-441d-9800-d1ce1c8f1f94-dbus\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869247 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-sysconfig\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869270 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99d8e137-a970-441d-9800-d1ce1c8f1f94-dbus\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869297 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-lib-modules\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869321 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-host\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869325 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-etc-sysconfig\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869345 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869370 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-etc-selinux\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869392 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99d8e137-a970-441d-9800-d1ce1c8f1f94-kubelet-config\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869409 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-host\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:13.869418 5084 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869413 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b599076-85db-4b98-8636-485e964a5e59-lib-modules\") pod \"tuned-pk5rg\" (UID: \"8b599076-85db-4b98-8636-485e964a5e59\") " pod="openshift-cluster-node-tuning-operator/tuned-pk5rg" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869428 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869431 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/412af756-be10-401f-8413-419d66bafaad-etc-selinux\") pod \"aws-ebs-csi-driver-node-lb8z6\" (UID: \"412af756-be10-401f-8413-419d66bafaad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lb8z6" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.869469 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99d8e137-a970-441d-9800-d1ce1c8f1f94-kubelet-config\") pod \"global-pull-secret-syncer-8wmmt\" (UID: \"99d8e137-a970-441d-9800-d1ce1c8f1f94\") " pod="kube-system/global-pull-secret-syncer-8wmmt" Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:13.869478 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm podName:9c059f8a-ed76-4ffb-9f24-be79941af384 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:14.369464308 +0000 UTC m=+1.963496274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vsnwk" (UID: "9c059f8a-ed76-4ffb-9f24-be79941af384") : configmap "kube-rbac-proxy" not found Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:13.869516 5084 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:07:13.869537 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:13.869549 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls podName:9c059f8a-ed76-4ffb-9f24-be79941af384 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:14.369540143 +0000 UTC m=+1.963572095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vsnwk" (UID: "9c059f8a-ed76-4ffb-9f24-be79941af384") : secret "insights-runtime-extractor-tls" not found Nov 03 22:07:13.871357 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.871339 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls\") pod \"image-registry-58d66d58db-phhsh\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:13.906114 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.906058 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:13.907176 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.907154 5084 scope.go:122] "RemoveContainer" containerID="6aa20fda0a2c8c0f59dd6e4890d8a0a2cd138f508306fdbc9ad380fa430cb880" Nov 03 22:07:13.916153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:13.916123 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:14.254016 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.253989 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w55xl_d3b08229-d363-4336-bb37-89df96b43dbe/serve-healthcheck-canary/0.log" Nov 03 22:07:14.274296 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.274253 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:14.274494 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.274302 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:14.274494 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.274406 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p6mft\" (UID: \"0a4cedce-72c5-4357-8001-23a5e14d7d9d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:14.274494 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:14.274429 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle podName:b537a9eb-5abf-4f6b-9a35-0f13c5211bf2 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:15.274411884 +0000 UTC m=+2.868443865 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle") pod "router-default-84d846b7f5-4qjqn" (UID: "b537a9eb-5abf-4f6b-9a35-0f13c5211bf2") : configmap references non-existent config key: service-ca.crt Nov 03 22:07:14.274494 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:14.274467 5084 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:07:14.274494 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:14.274483 5084 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:07:14.274691 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.274505 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:14.274691 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:14.274529 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls podName:f2a09669-a5d7-4dbb-9ddf-842f4f919028 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:15.274516674 +0000 UTC m=+2.868548662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dfffx" (UID: "f2a09669-a5d7-4dbb-9ddf-842f4f919028") : secret "samples-operator-tls" not found Nov 03 22:07:14.274691 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:14.274561 5084 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:14.274691 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:14.274580 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert podName:0a4cedce-72c5-4357-8001-23a5e14d7d9d nodeName:}" failed. No retries permitted until 2026-11-03 22:07:15.274568999 +0000 UTC m=+2.868600964 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p6mft" (UID: "0a4cedce-72c5-4357-8001-23a5e14d7d9d") : secret "networking-console-plugin-cert" not found Nov 03 22:07:14.274691 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:14.274599 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls podName:ab2929c7-ab47-4660-80eb-f973bf6efdae nodeName:}" failed. No retries permitted until 2026-11-03 22:07:15.274587606 +0000 UTC m=+2.868619583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8gtxf" (UID: "ab2929c7-ab47-4660-80eb-f973bf6efdae") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:14.375572 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.375522 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:14.375773 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.375638 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:07:14.375773 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.375670 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:14.375773 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:14.375698 5084 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:07:14.375773 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.375717 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:14.375773 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:14.375765 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls podName:9c059f8a-ed76-4ffb-9f24-be79941af384 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:15.375751409 +0000 UTC m=+2.969783365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vsnwk" (UID: "9c059f8a-ed76-4ffb-9f24-be79941af384") : secret "insights-runtime-extractor-tls" not found Nov 03 22:07:14.376012 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:14.375791 5084 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:07:14.376012 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:14.375856 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs podName:393f28cb-5fc8-4b3e-8852-eebe41b42a2b nodeName:}" failed. No retries permitted until 2026-11-03 22:07:15.375839163 +0000 UTC m=+2.969871142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs") pod "network-metrics-daemon-sfqbq" (UID: "393f28cb-5fc8-4b3e-8852-eebe41b42a2b") : secret "metrics-daemon-secret" not found Nov 03 22:07:14.376012 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:14.375795 5084 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:07:14.376012 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:14.375905 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls podName:43b5a34d-6354-4586-9dd1-55279e0dcfad nodeName:}" failed. No retries permitted until 2026-11-03 22:07:15.375895467 +0000 UTC m=+2.969927419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls") pod "dns-default-rntwm" (UID: "43b5a34d-6354-4586-9dd1-55279e0dcfad") : secret "dns-default-metrics-tls" not found Nov 03 22:07:14.376259 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.376239 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c059f8a-ed76-4ffb-9f24-be79941af384-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:14.387440 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.387411 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-595c4958c9-2rswm"] Nov 03 22:07:14.390760 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:14.390728 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0ac5b4_308e_4eca_8301_2c470cecdbef.slice/crio-3dc1a64919c777850103aa8417e1bf6f3859df6794a82a1e30c321176f98677c WatchSource:0}: Error finding container 3dc1a64919c777850103aa8417e1bf6f3859df6794a82a1e30c321176f98677c: Status 404 returned error can't find the container with id 3dc1a64919c777850103aa8417e1bf6f3859df6794a82a1e30c321176f98677c Nov 03 22:07:14.392154 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.392127 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58d66d58db-phhsh"] Nov 03 22:07:14.455546 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.455520 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-5nc9k_3da2c3b3-3a29-48e6-8281-8bdcd184b3f0/migrator/0.log" Nov 03 22:07:14.468752 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.468716 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:07:14.651350 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.651316 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-gpqdr_fdc0640a-6da9-4cd2-a6a8-07d56a91718e/migrator/0.log" Nov 03 22:07:14.713301 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.713253 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d66d58db-phhsh" event={"ID":"aacf527d-b30b-48cd-b5fd-be3eba9a2095","Type":"ContainerStarted","Data":"bc1086b45c9e8e98e78ef642018522560b24a9e63c7ce83bc58eea33de57d8f4"} Nov 03 22:07:14.713508 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.713353 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d66d58db-phhsh" event={"ID":"aacf527d-b30b-48cd-b5fd-be3eba9a2095","Type":"ContainerStarted","Data":"ab947b0374174d25f455fe14fc88896a77f5d9c7baefdd692b1a8382445f1b7f"} Nov 03 22:07:14.715342 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.715304 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-595c4958c9-2rswm" event={"ID":"7c0ac5b4-308e-4eca-8301-2c470cecdbef","Type":"ContainerStarted","Data":"979fcab098aadaa834c80b4103378a4b40e607569531d2d47a62979dd5acc7d1"} Nov 03 22:07:14.715514 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.715371 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-595c4958c9-2rswm" event={"ID":"7c0ac5b4-308e-4eca-8301-2c470cecdbef","Type":"ContainerStarted","Data":"3dc1a64919c777850103aa8417e1bf6f3859df6794a82a1e30c321176f98677c"} Nov 03 22:07:14.717646 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.717621 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:07:14.717781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.717756 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" event={"ID":"d1498363-2c98-4046-8feb-abb772a80149","Type":"ContainerStarted","Data":"db3c9e3dff43957cb9eb52b96ec1ed1b7ce92538c5d35e063d15901976ed68a6"} Nov 03 22:07:14.851767 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:14.851737 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z_e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3/kube-storage-version-migrator-operator/0.log" Nov 03 22:07:15.173242 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:15.173188 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:15.286862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:15.286816 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p6mft\" (UID: \"0a4cedce-72c5-4357-8001-23a5e14d7d9d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:15.287043 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:15.286906 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:15.287043 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:15.286963 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:15.287043 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:15.286975 5084 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:07:15.287043 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:15.286992 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:15.287262 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:15.287052 5084 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:15.287262 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:15.287056 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert podName:0a4cedce-72c5-4357-8001-23a5e14d7d9d nodeName:}" failed. No retries permitted until 2026-11-03 22:07:17.287035675 +0000 UTC m=+4.881067644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p6mft" (UID: "0a4cedce-72c5-4357-8001-23a5e14d7d9d") : secret "networking-console-plugin-cert" not found Nov 03 22:07:15.287262 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:15.287139 5084 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:07:15.287262 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:15.287159 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls podName:ab2929c7-ab47-4660-80eb-f973bf6efdae nodeName:}" failed. No retries permitted until 2026-11-03 22:07:17.287141332 +0000 UTC m=+4.881173298 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8gtxf" (UID: "ab2929c7-ab47-4660-80eb-f973bf6efdae") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:15.287262 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:15.287174 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle podName:b537a9eb-5abf-4f6b-9a35-0f13c5211bf2 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:17.287167657 +0000 UTC m=+4.881199613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle") pod "router-default-84d846b7f5-4qjqn" (UID: "b537a9eb-5abf-4f6b-9a35-0f13c5211bf2") : configmap references non-existent config key: service-ca.crt Nov 03 22:07:15.287262 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:15.287186 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls podName:f2a09669-a5d7-4dbb-9ddf-842f4f919028 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:17.287179858 +0000 UTC m=+4.881211810 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dfffx" (UID: "f2a09669-a5d7-4dbb-9ddf-842f4f919028") : secret "samples-operator-tls" not found Nov 03 22:07:15.388042 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:15.388004 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:07:15.388247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:15.388085 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:15.388247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:15.388167 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:15.388247 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:15.388190 5084 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:07:15.388413 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:15.388255 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs podName:393f28cb-5fc8-4b3e-8852-eebe41b42a2b nodeName:}" failed. No retries permitted until 2026-11-03 22:07:17.388238344 +0000 UTC m=+4.982270321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs") pod "network-metrics-daemon-sfqbq" (UID: "393f28cb-5fc8-4b3e-8852-eebe41b42a2b") : secret "metrics-daemon-secret" not found Nov 03 22:07:15.388413 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:15.388262 5084 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:07:15.388413 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:15.388295 5084 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:07:15.388413 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:15.388332 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls podName:43b5a34d-6354-4586-9dd1-55279e0dcfad nodeName:}" failed. No retries permitted until 2026-11-03 22:07:17.388311113 +0000 UTC m=+4.982343087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls") pod "dns-default-rntwm" (UID: "43b5a34d-6354-4586-9dd1-55279e0dcfad") : secret "dns-default-metrics-tls" not found Nov 03 22:07:15.388413 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:15.388354 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls podName:9c059f8a-ed76-4ffb-9f24-be79941af384 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:17.388343213 +0000 UTC m=+4.982375170 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vsnwk" (UID: "9c059f8a-ed76-4ffb-9f24-be79941af384") : secret "insights-runtime-extractor-tls" not found Nov 03 22:07:15.495515 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:15.495432 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:15.500396 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:15.500364 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9mgrk" Nov 03 22:07:15.721156 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:15.721125 5084 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:07:16.656302 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:16.656264 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:17.306757 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.306710 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:17.307307 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.306835 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p6mft\" (UID: \"0a4cedce-72c5-4357-8001-23a5e14d7d9d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:17.307307 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.306896 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:17.307307 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.306962 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:17.307307 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:17.306984 5084 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:07:17.307307 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:17.307092 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert podName:0a4cedce-72c5-4357-8001-23a5e14d7d9d nodeName:}" failed. No retries permitted until 2026-11-03 22:07:21.30705764 +0000 UTC m=+8.901089620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p6mft" (UID: "0a4cedce-72c5-4357-8001-23a5e14d7d9d") : secret "networking-console-plugin-cert" not found Nov 03 22:07:17.307307 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:17.307161 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle podName:b537a9eb-5abf-4f6b-9a35-0f13c5211bf2 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:21.307144203 +0000 UTC m=+8.901176169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle") pod "router-default-84d846b7f5-4qjqn" (UID: "b537a9eb-5abf-4f6b-9a35-0f13c5211bf2") : configmap references non-existent config key: service-ca.crt Nov 03 22:07:17.307307 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:17.307209 5084 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:17.307307 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:17.307273 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls podName:ab2929c7-ab47-4660-80eb-f973bf6efdae nodeName:}" failed. No retries permitted until 2026-11-03 22:07:21.307262689 +0000 UTC m=+8.901294668 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8gtxf" (UID: "ab2929c7-ab47-4660-80eb-f973bf6efdae") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:17.309372 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.309330 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2a09669-a5d7-4dbb-9ddf-842f4f919028-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dfffx\" (UID: \"f2a09669-a5d7-4dbb-9ddf-842f4f919028\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:17.407981 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.407935 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:17.408205 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.408028 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:17.408205 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.408149 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:07:17.408205 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:17.408169 5084 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:07:17.408361 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:17.408250 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls podName:9c059f8a-ed76-4ffb-9f24-be79941af384 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:21.408236056 +0000 UTC m=+9.002268040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vsnwk" (UID: "9c059f8a-ed76-4ffb-9f24-be79941af384") : secret "insights-runtime-extractor-tls" not found Nov 03 22:07:17.410365 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.410324 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43b5a34d-6354-4586-9dd1-55279e0dcfad-metrics-tls\") pod \"dns-default-rntwm\" (UID: \"43b5a34d-6354-4586-9dd1-55279e0dcfad\") " pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:17.410493 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.410469 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393f28cb-5fc8-4b3e-8852-eebe41b42a2b-metrics-certs\") pod \"network-metrics-daemon-sfqbq\" (UID: \"393f28cb-5fc8-4b3e-8852-eebe41b42a2b\") " pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:07:17.506406 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.506367 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" Nov 03 22:07:17.515916 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.515881 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sfqbq" Nov 03 22:07:17.518118 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.518045 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:17.730251 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.730221 5084 generic.go:356] "Generic (PLEG): container finished" podID="60e01c32-cde7-4c5e-add4-ca8c17e82e34" containerID="6d8b523fd09bacad32ffe9c4d48457dbd1a2efd39af5487f0eb68a70e6ae2a13" exitCode=0 Nov 03 22:07:17.730399 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.730277 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fqp5x" event={"ID":"60e01c32-cde7-4c5e-add4-ca8c17e82e34","Type":"ContainerDied","Data":"6d8b523fd09bacad32ffe9c4d48457dbd1a2efd39af5487f0eb68a70e6ae2a13"} Nov 03 22:07:17.730680 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.730664 5084 scope.go:122] "RemoveContainer" containerID="6d8b523fd09bacad32ffe9c4d48457dbd1a2efd39af5487f0eb68a70e6ae2a13" Nov 03 22:07:17.907962 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.907926 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx"] Nov 03 22:07:17.963579 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:17.963551 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393f28cb_5fc8_4b3e_8852_eebe41b42a2b.slice/crio-c853e319a0e12d01ef4619d068074e323c794ccf54d65aa8462c2ad48ba2f2ee WatchSource:0}: Error finding container c853e319a0e12d01ef4619d068074e323c794ccf54d65aa8462c2ad48ba2f2ee: Status 404 returned error can't find the container with id c853e319a0e12d01ef4619d068074e323c794ccf54d65aa8462c2ad48ba2f2ee Nov 03 22:07:17.966451 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.966432 5084 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:07:17.966552 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:17.966510 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sfqbq"] Nov 03 22:07:18.010794 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:18.010762 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rntwm"] Nov 03 22:07:18.013998 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:18.013968 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b5a34d_6354_4586_9dd1_55279e0dcfad.slice/crio-0610bbb1ca59fb91e4778f5b762815a0fbeb9663d56f6f993fa0bcc0505742c3 WatchSource:0}: Error finding container 0610bbb1ca59fb91e4778f5b762815a0fbeb9663d56f6f993fa0bcc0505742c3: Status 404 returned error can't find the container with id 0610bbb1ca59fb91e4778f5b762815a0fbeb9663d56f6f993fa0bcc0505742c3 Nov 03 22:07:18.716587 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:18.716556 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:18.735769 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:18.735736 5084 generic.go:356] "Generic (PLEG): container finished" podID="e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3" containerID="697b465a38cad9e9a0be32a78985bad7083c2b7d461e4706ae1bb573d45e6a2a" exitCode=0 Nov 03 22:07:18.735971 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:18.735816 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" event={"ID":"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3","Type":"ContainerDied","Data":"697b465a38cad9e9a0be32a78985bad7083c2b7d461e4706ae1bb573d45e6a2a"} Nov 03 22:07:18.736235 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:18.736219 5084 scope.go:122] "RemoveContainer" containerID="697b465a38cad9e9a0be32a78985bad7083c2b7d461e4706ae1bb573d45e6a2a" Nov 03 22:07:18.737225 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:18.737194 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" event={"ID":"f2a09669-a5d7-4dbb-9ddf-842f4f919028","Type":"ContainerStarted","Data":"70311e1fff56dd7d67fa2f5d0b392e09884d38ced5d0157ab60bb05918eca366"} Nov 03 22:07:18.738613 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:18.738589 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rntwm" event={"ID":"43b5a34d-6354-4586-9dd1-55279e0dcfad","Type":"ContainerStarted","Data":"0610bbb1ca59fb91e4778f5b762815a0fbeb9663d56f6f993fa0bcc0505742c3"} Nov 03 22:07:18.740823 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:18.740793 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fqp5x" event={"ID":"60e01c32-cde7-4c5e-add4-ca8c17e82e34","Type":"ContainerStarted","Data":"891ba1ef7bacb3217c63e98ad5d5f6779dc848c55c2aecc39bae8c04069083f1"} Nov 03 22:07:18.742207 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:18.742182 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sfqbq" event={"ID":"393f28cb-5fc8-4b3e-8852-eebe41b42a2b","Type":"ContainerStarted","Data":"c853e319a0e12d01ef4619d068074e323c794ccf54d65aa8462c2ad48ba2f2ee"} Nov 03 22:07:18.743778 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:18.743756 5084 generic.go:356] "Generic (PLEG): container finished" podID="e099b6bc-addb-428b-9a88-2c8b8e544f46" containerID="142d34367b57e97e74b5a76ee1fdd261c9a9aa7b2f000d7937fdbb0ceb3d7eb8" exitCode=0 Nov 03 22:07:18.743867 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:18.743791 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" event={"ID":"e099b6bc-addb-428b-9a88-2c8b8e544f46","Type":"ContainerDied","Data":"142d34367b57e97e74b5a76ee1fdd261c9a9aa7b2f000d7937fdbb0ceb3d7eb8"} Nov 03 22:07:18.744119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:18.744105 5084 scope.go:122] "RemoveContainer" containerID="142d34367b57e97e74b5a76ee1fdd261c9a9aa7b2f000d7937fdbb0ceb3d7eb8" Nov 03 22:07:19.752675 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:19.752633 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-96jwb" event={"ID":"e099b6bc-addb-428b-9a88-2c8b8e544f46","Type":"ContainerStarted","Data":"ea06ea28b73f6476513da82a9b8e25d37936bfe08c1e1299a68aad4df02fc8fc"} Nov 03 22:07:19.755532 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:19.755499 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-ndh4z" event={"ID":"e5a20614-9869-4b3f-8c5b-ef5ef6e28bd3","Type":"ContainerStarted","Data":"2a20dc9b1fa6f785a557c9c474f1e1effaf875575d9d9633c2d36cc7e5e18840"} Nov 03 22:07:19.859862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:19.859830 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:19.860044 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:19.860037 5084 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:07:20.761918 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:20.761819 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sfqbq" event={"ID":"393f28cb-5fc8-4b3e-8852-eebe41b42a2b","Type":"ContainerStarted","Data":"7992e120cd68b9f00554024cbc4e94765de9543de3aa8454a1b284c995fa458a"} Nov 03 22:07:20.761918 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:20.761881 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sfqbq" event={"ID":"393f28cb-5fc8-4b3e-8852-eebe41b42a2b","Type":"ContainerStarted","Data":"a888fb581dcdc1d8a07b0eeb67b065becf6e8dd35818df3eb5961a14b90d54f9"} Nov 03 22:07:21.349464 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:21.349357 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p6mft\" (UID: \"0a4cedce-72c5-4357-8001-23a5e14d7d9d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:21.349464 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:21.349441 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:21.349701 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:21.349491 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:21.349701 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:21.349610 5084 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:21.349701 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:21.349675 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls podName:ab2929c7-ab47-4660-80eb-f973bf6efdae nodeName:}" failed. No retries permitted until 2026-11-03 22:07:29.349655622 +0000 UTC m=+16.943687574 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8gtxf" (UID: "ab2929c7-ab47-4660-80eb-f973bf6efdae") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:07:21.349830 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:21.349704 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle podName:b537a9eb-5abf-4f6b-9a35-0f13c5211bf2 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:29.349684311 +0000 UTC m=+16.943716266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle") pod "router-default-84d846b7f5-4qjqn" (UID: "b537a9eb-5abf-4f6b-9a35-0f13c5211bf2") : configmap references non-existent config key: service-ca.crt Nov 03 22:07:21.352053 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:21.352022 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a4cedce-72c5-4357-8001-23a5e14d7d9d-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p6mft\" (UID: \"0a4cedce-72c5-4357-8001-23a5e14d7d9d\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:21.409828 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:21.409785 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" Nov 03 22:07:21.450962 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:21.450884 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:21.451215 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:21.451044 5084 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:07:21.451215 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:21.451152 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls podName:9c059f8a-ed76-4ffb-9f24-be79941af384 nodeName:}" failed. No retries permitted until 2026-11-03 22:07:29.451130064 +0000 UTC m=+17.045162039 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vsnwk" (UID: "9c059f8a-ed76-4ffb-9f24-be79941af384") : secret "insights-runtime-extractor-tls" not found Nov 03 22:07:21.768400 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:21.768360 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rntwm" event={"ID":"43b5a34d-6354-4586-9dd1-55279e0dcfad","Type":"ContainerStarted","Data":"a51166e7565ae1c3de6aff9802677f6389c8e5d76782fb66ab62e43ee7360678"} Nov 03 22:07:21.768885 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:21.768455 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rntwm" event={"ID":"43b5a34d-6354-4586-9dd1-55279e0dcfad","Type":"ContainerStarted","Data":"8e1ea1358d0ab0230a651a83cc1bf41e568ed6d392883712ce61b0e516a21618"} Nov 03 22:07:21.806344 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:21.806242 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-p6mft"] Nov 03 22:07:21.809364 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:21.809319 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4cedce_72c5_4357_8001_23a5e14d7d9d.slice/crio-61f1f7a8efe22b6effd7914a30a5b4168359b0b1eafc7612188ed09109bfb982 WatchSource:0}: Error finding container 61f1f7a8efe22b6effd7914a30a5b4168359b0b1eafc7612188ed09109bfb982: Status 404 returned error can't find the container with id 61f1f7a8efe22b6effd7914a30a5b4168359b0b1eafc7612188ed09109bfb982 Nov 03 22:07:22.772773 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:22.772731 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" event={"ID":"0a4cedce-72c5-4357-8001-23a5e14d7d9d","Type":"ContainerStarted","Data":"61f1f7a8efe22b6effd7914a30a5b4168359b0b1eafc7612188ed09109bfb982"} Nov 03 22:07:22.774753 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:22.774711 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" event={"ID":"f2a09669-a5d7-4dbb-9ddf-842f4f919028","Type":"ContainerStarted","Data":"eaee7bb37840a677d89c8aee41b664ebf0d299b3f95db8c53b5b87cd769dd295"} Nov 03 22:07:22.775052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:22.775033 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:22.775156 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:22.775095 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dfffx" event={"ID":"f2a09669-a5d7-4dbb-9ddf-842f4f919028","Type":"ContainerStarted","Data":"d3ca12d316664e0ab4dd5180307c335d4bb16a0f0f8720d545f790b9926c50ff"} Nov 03 22:07:23.582074 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:23.582020 5084 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4rnh_19a326ff-4502-4590-9c0d-194f8cf14abd/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:23.819864 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:23.819829 5084 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4rnh_19a326ff-4502-4590-9c0d-194f8cf14abd/konnectivity-agent/0.log" limit=1024 Nov 03 22:07:24.789216 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:24.789179 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-p6mft" event={"ID":"0a4cedce-72c5-4357-8001-23a5e14d7d9d","Type":"ContainerStarted","Data":"786fb37a54661e3f7dbed18845f66eedd6c841c55a2c07a473501c05c2a471f1"} Nov 03 22:07:27.353568 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:27.353532 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-4xsj8-t7xhr"] Nov 03 22:07:27.387942 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:27.387910 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" Nov 03 22:07:27.404990 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:27.404955 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-595c4958c9-2rswm"] Nov 03 22:07:27.460677 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:27.460645 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4xsj8-t7xhr"] Nov 03 22:07:27.504938 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:27.504866 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jxsb\" (UniqueName: \"kubernetes.io/projected/6884fe53-de40-4011-9f82-485876d56218-kube-api-access-4jxsb\") pod \"periodic-gathering-4xsj8-t7xhr\" (UID: \"6884fe53-de40-4011-9f82-485876d56218\") " pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" Nov 03 22:07:27.505163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:27.505022 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6884fe53-de40-4011-9f82-485876d56218-service-ca-bundle\") pod \"periodic-gathering-4xsj8-t7xhr\" (UID: \"6884fe53-de40-4011-9f82-485876d56218\") " pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" Nov 03 22:07:27.505163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:27.505101 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6884fe53-de40-4011-9f82-485876d56218-archives-path\") pod \"periodic-gathering-4xsj8-t7xhr\" (UID: \"6884fe53-de40-4011-9f82-485876d56218\") " pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" Nov 03 22:07:27.505308 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:27.505218 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6884fe53-de40-4011-9f82-485876d56218-trusted-ca-bundle\") pod \"periodic-gathering-4xsj8-t7xhr\" (UID: \"6884fe53-de40-4011-9f82-485876d56218\") " pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" Nov 03 22:07:27.606465 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:27.606372 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6884fe53-de40-4011-9f82-485876d56218-trusted-ca-bundle\") pod \"periodic-gathering-4xsj8-t7xhr\" (UID: \"6884fe53-de40-4011-9f82-485876d56218\") " pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" Nov 03 22:07:27.606465 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:27.606450 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jxsb\" (UniqueName: \"kubernetes.io/projected/6884fe53-de40-4011-9f82-485876d56218-kube-api-access-4jxsb\") pod \"periodic-gathering-4xsj8-t7xhr\" (UID: \"6884fe53-de40-4011-9f82-485876d56218\") " pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" Nov 03 22:07:27.606678 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:27.606483 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6884fe53-de40-4011-9f82-485876d56218-service-ca-bundle\") pod \"periodic-gathering-4xsj8-t7xhr\" (UID: \"6884fe53-de40-4011-9f82-485876d56218\") " pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" Nov 03 22:07:27.606678 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:27.606541 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6884fe53-de40-4011-9f82-485876d56218-archives-path\") pod \"periodic-gathering-4xsj8-t7xhr\" (UID: \"6884fe53-de40-4011-9f82-485876d56218\") " pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" Nov 03 22:07:27.606987 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:27.606965 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6884fe53-de40-4011-9f82-485876d56218-archives-path\") pod \"periodic-gathering-4xsj8-t7xhr\" (UID: \"6884fe53-de40-4011-9f82-485876d56218\") " pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" Nov 03 22:07:27.607295 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:27.607275 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6884fe53-de40-4011-9f82-485876d56218-service-ca-bundle\") pod \"periodic-gathering-4xsj8-t7xhr\" (UID: \"6884fe53-de40-4011-9f82-485876d56218\") " pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" Nov 03 22:07:27.607415 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:27.607396 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6884fe53-de40-4011-9f82-485876d56218-trusted-ca-bundle\") pod \"periodic-gathering-4xsj8-t7xhr\" (UID: \"6884fe53-de40-4011-9f82-485876d56218\") " pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" Nov 03 22:07:27.665792 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:27.665760 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jxsb\" (UniqueName: \"kubernetes.io/projected/6884fe53-de40-4011-9f82-485876d56218-kube-api-access-4jxsb\") pod \"periodic-gathering-4xsj8-t7xhr\" (UID: \"6884fe53-de40-4011-9f82-485876d56218\") " pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" Nov 03 22:07:27.712150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:27.712099 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" Nov 03 22:07:28.100041 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:28.099999 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4xsj8-t7xhr"] Nov 03 22:07:28.103573 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:28.103513 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6884fe53_de40_4011_9f82_485876d56218.slice/crio-9e3ce94f023d943a6f320f6c400a8cc10ae56f3aa02ea953c522d9cf7ecbcc3c WatchSource:0}: Error finding container 9e3ce94f023d943a6f320f6c400a8cc10ae56f3aa02ea953c522d9cf7ecbcc3c: Status 404 returned error can't find the container with id 9e3ce94f023d943a6f320f6c400a8cc10ae56f3aa02ea953c522d9cf7ecbcc3c Nov 03 22:07:28.806817 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:28.806780 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" event={"ID":"6884fe53-de40-4011-9f82-485876d56218","Type":"ContainerStarted","Data":"18dee5532db44e5169bfd67de9674ff8614ffcadb3f350ea2af54322c700a2a6"} Nov 03 22:07:28.807231 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:28.806829 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" event={"ID":"6884fe53-de40-4011-9f82-485876d56218","Type":"ContainerStarted","Data":"9e3ce94f023d943a6f320f6c400a8cc10ae56f3aa02ea953c522d9cf7ecbcc3c"} Nov 03 22:07:28.921466 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:28.921404 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" podStartSLOduration=1.921388582 podStartE2EDuration="1.921388582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:28.921122733 +0000 UTC m=+16.515154708" watchObservedRunningTime="2026-11-03 22:07:28.921388582 +0000 UTC m=+16.515420550" Nov 03 22:07:29.420950 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:29.420895 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:29.421142 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:29.420972 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:29.421679 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:29.421648 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b537a9eb-5abf-4f6b-9a35-0f13c5211bf2-service-ca-bundle\") pod \"router-default-84d846b7f5-4qjqn\" (UID: \"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2\") " pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:29.423522 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:29.423499 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab2929c7-ab47-4660-80eb-f973bf6efdae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8gtxf\" (UID: \"ab2929c7-ab47-4660-80eb-f973bf6efdae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:29.504476 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:29.504442 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" Nov 03 22:07:29.504815 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:29.504779 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:29.522250 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:29.522215 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:29.524732 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:29.524702 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c059f8a-ed76-4ffb-9f24-be79941af384-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vsnwk\" (UID: \"9c059f8a-ed76-4ffb-9f24-be79941af384\") " pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:29.818647 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:29.818613 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vsnwk" Nov 03 22:07:29.942886 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:29.942849 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf"] Nov 03 22:07:29.946086 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:29.946034 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab2929c7_ab47_4660_80eb_f973bf6efdae.slice/crio-9eeac3176a28f32f28136cf24d0789252b3db7ab76c193ebc676ecca9e937a91 WatchSource:0}: Error finding container 9eeac3176a28f32f28136cf24d0789252b3db7ab76c193ebc676ecca9e937a91: Status 404 returned error can't find the container with id 9eeac3176a28f32f28136cf24d0789252b3db7ab76c193ebc676ecca9e937a91 Nov 03 22:07:29.952675 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:29.952643 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rntwm_43b5a34d-6354-4586-9dd1-55279e0dcfad/dns/0.log" Nov 03 22:07:29.991306 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:29.991261 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb537a9eb_5abf_4f6b_9a35_0f13c5211bf2.slice/crio-da0faed58cde97f18b525c1d0b6e268a0a92d6b73f58e1ba1e84daa9cd5935b8 WatchSource:0}: Error finding container da0faed58cde97f18b525c1d0b6e268a0a92d6b73f58e1ba1e84daa9cd5935b8: Status 404 returned error can't find the container with id da0faed58cde97f18b525c1d0b6e268a0a92d6b73f58e1ba1e84daa9cd5935b8 Nov 03 22:07:29.994220 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:29.994131 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-84d846b7f5-4qjqn"] Nov 03 22:07:30.098725 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:30.098695 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rntwm_43b5a34d-6354-4586-9dd1-55279e0dcfad/kube-rbac-proxy/0.log" Nov 03 22:07:30.221662 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:30.221593 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vsnwk"] Nov 03 22:07:30.225197 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:30.225166 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c059f8a_ed76_4ffb_9f24_be79941af384.slice/crio-4cb94cc2c1a9dd379c9215e239a1703acdb280163e4cf563f7c1acb5eade933a WatchSource:0}: Error finding container 4cb94cc2c1a9dd379c9215e239a1703acdb280163e4cf563f7c1acb5eade933a: Status 404 returned error can't find the container with id 4cb94cc2c1a9dd379c9215e239a1703acdb280163e4cf563f7c1acb5eade933a Nov 03 22:07:30.816470 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:30.816431 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-84d846b7f5-4qjqn" event={"ID":"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2","Type":"ContainerStarted","Data":"1460b7f2b1f2265c0d3db0ebd1f059cec102f47087d0cc6dd6198330a45e95d9"} Nov 03 22:07:30.816682 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:30.816484 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-84d846b7f5-4qjqn" event={"ID":"b537a9eb-5abf-4f6b-9a35-0f13c5211bf2","Type":"ContainerStarted","Data":"da0faed58cde97f18b525c1d0b6e268a0a92d6b73f58e1ba1e84daa9cd5935b8"} Nov 03 22:07:30.817872 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:30.817841 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" event={"ID":"ab2929c7-ab47-4660-80eb-f973bf6efdae","Type":"ContainerStarted","Data":"9eeac3176a28f32f28136cf24d0789252b3db7ab76c193ebc676ecca9e937a91"} Nov 03 22:07:30.819378 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:30.819350 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vsnwk" event={"ID":"9c059f8a-ed76-4ffb-9f24-be79941af384","Type":"ContainerStarted","Data":"593079d9b02eaf134aa5f3f7869bb576c41b6b311da2aeba4d0925f2dafc8b65"} Nov 03 22:07:30.819691 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:30.819399 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vsnwk" event={"ID":"9c059f8a-ed76-4ffb-9f24-be79941af384","Type":"ContainerStarted","Data":"4cb94cc2c1a9dd379c9215e239a1703acdb280163e4cf563f7c1acb5eade933a"} Nov 03 22:07:30.897228 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:30.897197 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-79sq5_866a1e7a-4728-4bd5-a88e-1a97f36f1eac/dns-node-resolver/0.log" Nov 03 22:07:31.267867 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:31.267821 5084 patch_prober.go:29] interesting pod/image-registry-58d66d58db-phhsh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 22:07:31.268050 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:31.267892 5084 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-58d66d58db-phhsh" podUID="aacf527d-b30b-48cd-b5fd-be3eba9a2095" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:07:31.298383 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:31.298355 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-58d66d58db-phhsh_aacf527d-b30b-48cd-b5fd-be3eba9a2095/registry/0.log" Nov 03 22:07:31.497933 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:31.497896 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-595c4958c9-2rswm_7c0ac5b4-308e-4eca-8301-2c470cecdbef/registry/0.log" Nov 03 22:07:31.505666 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:31.505635 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:31.508851 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:31.508822 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:31.824625 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:31.824586 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:31.826049 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:31.826020 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-84d846b7f5-4qjqn" Nov 03 22:07:32.097172 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:32.097077 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t6b4b_95246059-e5c1-4b42-9678-a1c8b2f63e80/node-ca/0.log" Nov 03 22:07:32.497287 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:32.497171 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84d846b7f5-4qjqn_b537a9eb-5abf-4f6b-9a35-0f13c5211bf2/router/0.log" Nov 03 22:07:32.786242 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:32.786194 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rntwm" Nov 03 22:07:32.832764 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:32.832727 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vsnwk" event={"ID":"9c059f8a-ed76-4ffb-9f24-be79941af384","Type":"ContainerStarted","Data":"95d85f9f55cded12d20e863f9c1cf0d686249f7ad7eb85b5e427a4bc24774601"} Nov 03 22:07:32.834584 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:32.834553 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8gtxf" event={"ID":"ab2929c7-ab47-4660-80eb-f973bf6efdae","Type":"ContainerStarted","Data":"804b30bdd46107e71ae047bb4322bf07515a410fe1365c075c78632e1907624c"} Nov 03 22:07:32.896890 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:32.896858 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w55xl_d3b08229-d363-4336-bb37-89df96b43dbe/serve-healthcheck-canary/0.log" Nov 03 22:07:33.679337 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:33.679299 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-78tk2" Nov 03 22:07:34.845797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:34.845749 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vsnwk" event={"ID":"9c059f8a-ed76-4ffb-9f24-be79941af384","Type":"ContainerStarted","Data":"efa4034e9df5fc3c370b3363473b58739b132f9272945650e1b147366a35adc5"} Nov 03 22:07:36.729287 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:36.729256 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:07:37.290743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:37.290709 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k4rnh" Nov 03 22:07:37.410620 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:37.410587 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:42.021192 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:42.021152 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:07:44.277933 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:44.277897 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58d66d58db-phhsh"] Nov 03 22:07:44.720961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:44.720889 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z4ct8" Nov 03 22:07:45.469914 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.469873 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xcffl"] Nov 03 22:07:45.473244 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.473222 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.526179 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.526141 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 22:07:45.526179 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.526142 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 22:07:45.568734 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.568689 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9jzv\" (UniqueName: \"kubernetes.io/projected/36630490-7771-45f7-af2a-5d37a089f9e9-kube-api-access-p9jzv\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.569009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.568741 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/36630490-7771-45f7-af2a-5d37a089f9e9-node-exporter-accelerators-collector-config\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.569009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.568831 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36630490-7771-45f7-af2a-5d37a089f9e9-sys\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.569009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.568887 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36630490-7771-45f7-af2a-5d37a089f9e9-metrics-client-ca\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.569009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.568994 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/36630490-7771-45f7-af2a-5d37a089f9e9-node-exporter-wtmp\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.569281 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.569124 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/36630490-7771-45f7-af2a-5d37a089f9e9-node-exporter-textfile\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.569281 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.569159 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/36630490-7771-45f7-af2a-5d37a089f9e9-node-exporter-tls\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.569281 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.569197 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/36630490-7771-45f7-af2a-5d37a089f9e9-root\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.569387 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.569278 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36630490-7771-45f7-af2a-5d37a089f9e9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.577308 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.577276 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 22:07:45.577453 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.577367 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xzmgl\"" Nov 03 22:07:45.577511 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.577484 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 22:07:45.670147 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.670106 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/36630490-7771-45f7-af2a-5d37a089f9e9-node-exporter-tls\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.670356 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.670155 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/36630490-7771-45f7-af2a-5d37a089f9e9-root\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.670356 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.670186 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36630490-7771-45f7-af2a-5d37a089f9e9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.670356 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.670232 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9jzv\" (UniqueName: \"kubernetes.io/projected/36630490-7771-45f7-af2a-5d37a089f9e9-kube-api-access-p9jzv\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.670356 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.670263 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/36630490-7771-45f7-af2a-5d37a089f9e9-node-exporter-accelerators-collector-config\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.670356 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.670270 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/36630490-7771-45f7-af2a-5d37a089f9e9-root\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.670356 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.670291 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36630490-7771-45f7-af2a-5d37a089f9e9-sys\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.670356 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.670334 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36630490-7771-45f7-af2a-5d37a089f9e9-sys\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.670748 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.670363 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36630490-7771-45f7-af2a-5d37a089f9e9-metrics-client-ca\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.670748 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.670409 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/36630490-7771-45f7-af2a-5d37a089f9e9-node-exporter-wtmp\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.670748 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.670472 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/36630490-7771-45f7-af2a-5d37a089f9e9-node-exporter-textfile\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.670748 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.670630 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/36630490-7771-45f7-af2a-5d37a089f9e9-node-exporter-wtmp\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.670900 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.670837 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/36630490-7771-45f7-af2a-5d37a089f9e9-node-exporter-textfile\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.670994 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.670968 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/36630490-7771-45f7-af2a-5d37a089f9e9-node-exporter-accelerators-collector-config\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.671041 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.671018 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36630490-7771-45f7-af2a-5d37a089f9e9-metrics-client-ca\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.672582 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.672561 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/36630490-7771-45f7-af2a-5d37a089f9e9-node-exporter-tls\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.672693 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.672637 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36630490-7771-45f7-af2a-5d37a089f9e9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.733487 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.733403 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9jzv\" (UniqueName: \"kubernetes.io/projected/36630490-7771-45f7-af2a-5d37a089f9e9-kube-api-access-p9jzv\") pod \"node-exporter-xcffl\" (UID: \"36630490-7771-45f7-af2a-5d37a089f9e9\") " pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.787321 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.787283 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xcffl" Nov 03 22:07:45.799041 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:07:45.799005 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36630490_7771_45f7_af2a_5d37a089f9e9.slice/crio-cacdaff2e26e54abce2bc30d387d75de9386339e0676817f96d3218429346a29 WatchSource:0}: Error finding container cacdaff2e26e54abce2bc30d387d75de9386339e0676817f96d3218429346a29: Status 404 returned error can't find the container with id cacdaff2e26e54abce2bc30d387d75de9386339e0676817f96d3218429346a29 Nov 03 22:07:45.892087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:45.892027 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xcffl" event={"ID":"36630490-7771-45f7-af2a-5d37a089f9e9","Type":"ContainerStarted","Data":"cacdaff2e26e54abce2bc30d387d75de9386339e0676817f96d3218429346a29"} Nov 03 22:07:47.902670 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:47.902633 5084 generic.go:356] "Generic (PLEG): container finished" podID="36630490-7771-45f7-af2a-5d37a089f9e9" containerID="69b259e952940424e479e345dcf41a5973ed38f6f381695f58cb8cc0a31c1959" exitCode=0 Nov 03 22:07:47.903183 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:47.902711 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xcffl" event={"ID":"36630490-7771-45f7-af2a-5d37a089f9e9","Type":"ContainerDied","Data":"69b259e952940424e479e345dcf41a5973ed38f6f381695f58cb8cc0a31c1959"} Nov 03 22:07:48.910640 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:48.910601 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xcffl" event={"ID":"36630490-7771-45f7-af2a-5d37a089f9e9","Type":"ContainerStarted","Data":"d95094e57b647ad2f5657e667f874ea69bb356f7d17b3fe1eb281a0cf4d9b531"} Nov 03 22:07:48.911097 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:48.910666 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xcffl" event={"ID":"36630490-7771-45f7-af2a-5d37a089f9e9","Type":"ContainerStarted","Data":"28df0cf438fea071675a5afd28bb4dab6b15043bbb2aa38df0eb1504331e88e9"} Nov 03 22:07:49.080158 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:49.080106 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xcffl" podStartSLOduration=2.386009353 podStartE2EDuration="4.080089313s" totalImagesPullingTime="1.623945369s" totalInitContainerRuntime="70.134591ms" isStatefulPod=true podCreationTimestamp="2026-11-03 22:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:07:49.079015805 +0000 UTC m=+36.673047782" watchObservedRunningTime="2026-11-03 22:07:49.080089313 +0000 UTC m=+36.674121287" Nov 03 22:07:52.426600 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.426502 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-595c4958c9-2rswm" podUID="7c0ac5b4-308e-4eca-8301-2c470cecdbef" containerName="registry" containerID="cri-o://979fcab098aadaa834c80b4103378a4b40e607569531d2d47a62979dd5acc7d1" gracePeriod=30 Nov 03 22:07:52.675658 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.675628 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:52.836791 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.836752 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-kube-api-access-qgntw\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-kube-api-access-qgntw\") pod \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " Nov 03 22:07:52.837023 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.836810 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-trusted-ca\") pod \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " Nov 03 22:07:52.837023 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.836848 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-bound-sa-token\") pod \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " Nov 03 22:07:52.837023 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.836908 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c0ac5b4-308e-4eca-8301-2c470cecdbef-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c0ac5b4-308e-4eca-8301-2c470cecdbef-ca-trust-extracted\") pod \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " Nov 03 22:07:52.837023 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.836971 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-certificates\") pod \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " Nov 03 22:07:52.837023 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.837016 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls\") pod \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " Nov 03 22:07:52.837323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.837046 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-installation-pull-secrets\") pod \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " Nov 03 22:07:52.837323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.837108 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-image-registry-private-configuration\") pod \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\" (UID: \"7c0ac5b4-308e-4eca-8301-2c470cecdbef\") " Nov 03 22:07:52.837622 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.837356 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-trusted-ca" pod "7c0ac5b4-308e-4eca-8301-2c470cecdbef" (UID: "7c0ac5b4-308e-4eca-8301-2c470cecdbef"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:07:52.838830 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.838792 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-certificates" pod "7c0ac5b4-308e-4eca-8301-2c470cecdbef" (UID: "7c0ac5b4-308e-4eca-8301-2c470cecdbef"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:07:52.839591 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.839541 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-bound-sa-token" pod "7c0ac5b4-308e-4eca-8301-2c470cecdbef" (UID: "7c0ac5b4-308e-4eca-8301-2c470cecdbef"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:07:52.839721 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.839689 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-kube-api-access-qgntw" pod "7c0ac5b4-308e-4eca-8301-2c470cecdbef" (UID: "7c0ac5b4-308e-4eca-8301-2c470cecdbef"). InnerVolumeSpecName "kube-api-access-qgntw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:07:52.842816 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.842776 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-installation-pull-secrets" pod "7c0ac5b4-308e-4eca-8301-2c470cecdbef" (UID: "7c0ac5b4-308e-4eca-8301-2c470cecdbef"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:07:52.850578 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.850518 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls" pod "7c0ac5b4-308e-4eca-8301-2c470cecdbef" (UID: "7c0ac5b4-308e-4eca-8301-2c470cecdbef"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:07:52.850734 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.850543 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-image-registry-private-configuration" pod "7c0ac5b4-308e-4eca-8301-2c470cecdbef" (UID: "7c0ac5b4-308e-4eca-8301-2c470cecdbef"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:07:52.851152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.851115 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c0ac5b4-308e-4eca-8301-2c470cecdbef-ca-trust-extracted" pod "7c0ac5b4-308e-4eca-8301-2c470cecdbef" (UID: "7c0ac5b4-308e-4eca-8301-2c470cecdbef"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:52.928475 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.928441 5084 generic.go:356] "Generic (PLEG): container finished" podID="7c0ac5b4-308e-4eca-8301-2c470cecdbef" containerID="979fcab098aadaa834c80b4103378a4b40e607569531d2d47a62979dd5acc7d1" exitCode=0 Nov 03 22:07:52.928660 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.928496 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-595c4958c9-2rswm" event={"ID":"7c0ac5b4-308e-4eca-8301-2c470cecdbef","Type":"ContainerDied","Data":"979fcab098aadaa834c80b4103378a4b40e607569531d2d47a62979dd5acc7d1"} Nov 03 22:07:52.928660 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.928502 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-595c4958c9-2rswm" Nov 03 22:07:52.928660 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.928560 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-595c4958c9-2rswm" event={"ID":"7c0ac5b4-308e-4eca-8301-2c470cecdbef","Type":"ContainerDied","Data":"3dc1a64919c777850103aa8417e1bf6f3859df6794a82a1e30c321176f98677c"} Nov 03 22:07:52.928660 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.928579 5084 scope.go:122] "RemoveContainer" containerID="979fcab098aadaa834c80b4103378a4b40e607569531d2d47a62979dd5acc7d1" Nov 03 22:07:52.938385 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.938361 5084 scope.go:122] "RemoveContainer" containerID="979fcab098aadaa834c80b4103378a4b40e607569531d2d47a62979dd5acc7d1" Nov 03 22:07:52.938767 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:52.938745 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"979fcab098aadaa834c80b4103378a4b40e607569531d2d47a62979dd5acc7d1\": container with ID starting with 979fcab098aadaa834c80b4103378a4b40e607569531d2d47a62979dd5acc7d1 not found: ID does not exist" containerID="979fcab098aadaa834c80b4103378a4b40e607569531d2d47a62979dd5acc7d1" Nov 03 22:07:52.938837 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.938783 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979fcab098aadaa834c80b4103378a4b40e607569531d2d47a62979dd5acc7d1"} err="failed to get container status \"979fcab098aadaa834c80b4103378a4b40e607569531d2d47a62979dd5acc7d1\": rpc error: code = NotFound desc = could not find container \"979fcab098aadaa834c80b4103378a4b40e607569531d2d47a62979dd5acc7d1\": container with ID starting with 979fcab098aadaa834c80b4103378a4b40e607569531d2d47a62979dd5acc7d1 not found: ID does not exist" Nov 03 22:07:52.940196 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.940177 5084 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-installation-pull-secrets\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:52.940250 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.940202 5084 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c0ac5b4-308e-4eca-8301-2c470cecdbef-image-registry-private-configuration\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:52.940250 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.940214 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgntw\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-kube-api-access-qgntw\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:52.940250 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.940226 5084 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-trusted-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:52.940250 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.940235 5084 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-bound-sa-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:52.940250 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.940244 5084 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c0ac5b4-308e-4eca-8301-2c470cecdbef-ca-trust-extracted\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:52.940392 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.940252 5084 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-certificates\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:52.940392 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:52.940261 5084 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c0ac5b4-308e-4eca-8301-2c470cecdbef-registry-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:53.095906 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:53.095816 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-595c4958c9-2rswm"] Nov 03 22:07:53.099386 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:53.099357 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-595c4958c9-2rswm"] Nov 03 22:07:54.626514 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:54.626475 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c0ac5b4-308e-4eca-8301-2c470cecdbef" path="/var/lib/kubelet/pods/7c0ac5b4-308e-4eca-8301-2c470cecdbef/volumes" Nov 03 22:07:59.251570 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.251535 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-wtfch-phz6j"] Nov 03 22:07:59.252161 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.251743 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-wtfch-phz6j" podUID="19c22c7c-c7e9-4ef8-9e55-245393662f36" containerName="insights-gathering" containerID="cri-o://f63d3a0316a67bf8071ed5a4c080f5d6f34bc8778614a7f6e6db6515439d7315" gracePeriod=30 Nov 03 22:07:59.519987 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.519957 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:59.700606 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.700563 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-trusted-ca-bundle\") pod \"19c22c7c-c7e9-4ef8-9e55-245393662f36\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " Nov 03 22:07:59.700782 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.700683 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-service-ca-bundle\") pod \"19c22c7c-c7e9-4ef8-9e55-245393662f36\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " Nov 03 22:07:59.700782 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.700708 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c22c7c-c7e9-4ef8-9e55-245393662f36-kube-api-access-wzpgz\" (UniqueName: \"kubernetes.io/projected/19c22c7c-c7e9-4ef8-9e55-245393662f36-kube-api-access-wzpgz\") pod \"19c22c7c-c7e9-4ef8-9e55-245393662f36\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " Nov 03 22:07:59.700782 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.700738 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c22c7c-c7e9-4ef8-9e55-245393662f36-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/19c22c7c-c7e9-4ef8-9e55-245393662f36-archives-path\") pod \"19c22c7c-c7e9-4ef8-9e55-245393662f36\" (UID: \"19c22c7c-c7e9-4ef8-9e55-245393662f36\") " Nov 03 22:07:59.701038 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.701004 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-trusted-ca-bundle" pod "19c22c7c-c7e9-4ef8-9e55-245393662f36" (UID: "19c22c7c-c7e9-4ef8-9e55-245393662f36"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:07:59.701038 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.701027 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-service-ca-bundle" pod "19c22c7c-c7e9-4ef8-9e55-245393662f36" (UID: "19c22c7c-c7e9-4ef8-9e55-245393662f36"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:07:59.701173 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.701134 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c22c7c-c7e9-4ef8-9e55-245393662f36-archives-path" pod "19c22c7c-c7e9-4ef8-9e55-245393662f36" (UID: "19c22c7c-c7e9-4ef8-9e55-245393662f36"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:07:59.703054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.703030 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c22c7c-c7e9-4ef8-9e55-245393662f36-kube-api-access-wzpgz" pod "19c22c7c-c7e9-4ef8-9e55-245393662f36" (UID: "19c22c7c-c7e9-4ef8-9e55-245393662f36"). InnerVolumeSpecName "kube-api-access-wzpgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:07:59.801307 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.801270 5084 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/19c22c7c-c7e9-4ef8-9e55-245393662f36-archives-path\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:59.801307 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.801306 5084 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-trusted-ca-bundle\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:59.801556 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.801319 5084 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19c22c7c-c7e9-4ef8-9e55-245393662f36-service-ca-bundle\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:59.801556 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.801332 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzpgz\" (UniqueName: \"kubernetes.io/projected/19c22c7c-c7e9-4ef8-9e55-245393662f36-kube-api-access-wzpgz\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:07:59.964727 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.964693 5084 generic.go:356] "Generic (PLEG): container finished" podID="19c22c7c-c7e9-4ef8-9e55-245393662f36" containerID="f63d3a0316a67bf8071ed5a4c080f5d6f34bc8778614a7f6e6db6515439d7315" exitCode=2 Nov 03 22:07:59.964928 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.964735 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wtfch-phz6j" Nov 03 22:07:59.964928 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.964774 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wtfch-phz6j" event={"ID":"19c22c7c-c7e9-4ef8-9e55-245393662f36","Type":"ContainerDied","Data":"f63d3a0316a67bf8071ed5a4c080f5d6f34bc8778614a7f6e6db6515439d7315"} Nov 03 22:07:59.964928 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.964827 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wtfch-phz6j" event={"ID":"19c22c7c-c7e9-4ef8-9e55-245393662f36","Type":"ContainerDied","Data":"4dbbb34e777a262f3c6c8d1354e17613d9b1829a51059db8c524f77e2fa72386"} Nov 03 22:07:59.964928 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.964847 5084 scope.go:122] "RemoveContainer" containerID="f63d3a0316a67bf8071ed5a4c080f5d6f34bc8778614a7f6e6db6515439d7315" Nov 03 22:07:59.974424 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.974401 5084 scope.go:122] "RemoveContainer" containerID="f63d3a0316a67bf8071ed5a4c080f5d6f34bc8778614a7f6e6db6515439d7315" Nov 03 22:07:59.974781 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:07:59.974761 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63d3a0316a67bf8071ed5a4c080f5d6f34bc8778614a7f6e6db6515439d7315\": container with ID starting with f63d3a0316a67bf8071ed5a4c080f5d6f34bc8778614a7f6e6db6515439d7315 not found: ID does not exist" containerID="f63d3a0316a67bf8071ed5a4c080f5d6f34bc8778614a7f6e6db6515439d7315" Nov 03 22:07:59.974851 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:07:59.974793 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63d3a0316a67bf8071ed5a4c080f5d6f34bc8778614a7f6e6db6515439d7315"} err="failed to get container status \"f63d3a0316a67bf8071ed5a4c080f5d6f34bc8778614a7f6e6db6515439d7315\": rpc error: code = NotFound desc = could not find container \"f63d3a0316a67bf8071ed5a4c080f5d6f34bc8778614a7f6e6db6515439d7315\": container with ID starting with f63d3a0316a67bf8071ed5a4c080f5d6f34bc8778614a7f6e6db6515439d7315 not found: ID does not exist" Nov 03 22:08:00.134528 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:00.134422 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-wtfch-phz6j"] Nov 03 22:08:00.185363 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:00.185330 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-wtfch-phz6j"] Nov 03 22:08:00.626627 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:00.626592 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c22c7c-c7e9-4ef8-9e55-245393662f36" path="/var/lib/kubelet/pods/19c22c7c-c7e9-4ef8-9e55-245393662f36/volumes" Nov 03 22:08:09.298890 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.298823 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-58d66d58db-phhsh" podUID="aacf527d-b30b-48cd-b5fd-be3eba9a2095" containerName="registry" containerID="cri-o://bc1086b45c9e8e98e78ef642018522560b24a9e63c7ce83bc58eea33de57d8f4" gracePeriod=30 Nov 03 22:08:09.563549 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.563471 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:08:09.581474 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.581439 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-image-registry-private-configuration\") pod \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " Nov 03 22:08:09.581668 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.581497 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-bound-sa-token\") pod \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " Nov 03 22:08:09.581668 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.581535 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-certificates\") pod \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " Nov 03 22:08:09.581668 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.581561 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-trusted-ca\") pod \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " Nov 03 22:08:09.581668 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.581587 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-installation-pull-secrets\") pod \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " Nov 03 22:08:09.581668 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.581628 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls\") pod \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " Nov 03 22:08:09.581966 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.581933 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-certificates" pod "aacf527d-b30b-48cd-b5fd-be3eba9a2095" (UID: "aacf527d-b30b-48cd-b5fd-be3eba9a2095"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:09.582042 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.582015 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-trusted-ca" pod "aacf527d-b30b-48cd-b5fd-be3eba9a2095" (UID: "aacf527d-b30b-48cd-b5fd-be3eba9a2095"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:09.584199 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.584157 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-installation-pull-secrets" pod "aacf527d-b30b-48cd-b5fd-be3eba9a2095" (UID: "aacf527d-b30b-48cd-b5fd-be3eba9a2095"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:09.584318 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.584231 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls" pod "aacf527d-b30b-48cd-b5fd-be3eba9a2095" (UID: "aacf527d-b30b-48cd-b5fd-be3eba9a2095"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:09.584318 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.584245 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-image-registry-private-configuration" pod "aacf527d-b30b-48cd-b5fd-be3eba9a2095" (UID: "aacf527d-b30b-48cd-b5fd-be3eba9a2095"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:08:09.584318 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.584305 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-bound-sa-token" pod "aacf527d-b30b-48cd-b5fd-be3eba9a2095" (UID: "aacf527d-b30b-48cd-b5fd-be3eba9a2095"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:09.682558 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.682519 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-kube-api-access-ms2cm\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-kube-api-access-ms2cm\") pod \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " Nov 03 22:08:09.682761 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.682571 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacf527d-b30b-48cd-b5fd-be3eba9a2095-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aacf527d-b30b-48cd-b5fd-be3eba9a2095-ca-trust-extracted\") pod \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\" (UID: \"aacf527d-b30b-48cd-b5fd-be3eba9a2095\") " Nov 03 22:08:09.682761 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.682687 5084 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-bound-sa-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.682761 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.682699 5084 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-certificates\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.682761 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.682710 5084 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aacf527d-b30b-48cd-b5fd-be3eba9a2095-trusted-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.682761 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.682720 5084 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-installation-pull-secrets\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.682761 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.682729 5084 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-registry-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.682761 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.682738 5084 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aacf527d-b30b-48cd-b5fd-be3eba9a2095-image-registry-private-configuration\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.684811 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.684778 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-kube-api-access-ms2cm" pod "aacf527d-b30b-48cd-b5fd-be3eba9a2095" (UID: "aacf527d-b30b-48cd-b5fd-be3eba9a2095"). InnerVolumeSpecName "kube-api-access-ms2cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:09.688767 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.688736 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacf527d-b30b-48cd-b5fd-be3eba9a2095-ca-trust-extracted" pod "aacf527d-b30b-48cd-b5fd-be3eba9a2095" (UID: "aacf527d-b30b-48cd-b5fd-be3eba9a2095"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:09.783400 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.783356 5084 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aacf527d-b30b-48cd-b5fd-be3eba9a2095-ca-trust-extracted\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:09.783400 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:09.783399 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms2cm\" (UniqueName: \"kubernetes.io/projected/aacf527d-b30b-48cd-b5fd-be3eba9a2095-kube-api-access-ms2cm\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:10.009773 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:10.009675 5084 generic.go:356] "Generic (PLEG): container finished" podID="aacf527d-b30b-48cd-b5fd-be3eba9a2095" containerID="bc1086b45c9e8e98e78ef642018522560b24a9e63c7ce83bc58eea33de57d8f4" exitCode=0 Nov 03 22:08:10.009773 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:10.009722 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d66d58db-phhsh" Nov 03 22:08:10.009773 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:10.009754 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d66d58db-phhsh" event={"ID":"aacf527d-b30b-48cd-b5fd-be3eba9a2095","Type":"ContainerDied","Data":"bc1086b45c9e8e98e78ef642018522560b24a9e63c7ce83bc58eea33de57d8f4"} Nov 03 22:08:10.010022 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:10.009808 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58d66d58db-phhsh" event={"ID":"aacf527d-b30b-48cd-b5fd-be3eba9a2095","Type":"ContainerDied","Data":"ab947b0374174d25f455fe14fc88896a77f5d9c7baefdd692b1a8382445f1b7f"} Nov 03 22:08:10.010022 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:10.009840 5084 scope.go:122] "RemoveContainer" containerID="bc1086b45c9e8e98e78ef642018522560b24a9e63c7ce83bc58eea33de57d8f4" Nov 03 22:08:10.019607 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:10.019584 5084 scope.go:122] "RemoveContainer" containerID="bc1086b45c9e8e98e78ef642018522560b24a9e63c7ce83bc58eea33de57d8f4" Nov 03 22:08:10.019939 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:08:10.019916 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc1086b45c9e8e98e78ef642018522560b24a9e63c7ce83bc58eea33de57d8f4\": container with ID starting with bc1086b45c9e8e98e78ef642018522560b24a9e63c7ce83bc58eea33de57d8f4 not found: ID does not exist" containerID="bc1086b45c9e8e98e78ef642018522560b24a9e63c7ce83bc58eea33de57d8f4" Nov 03 22:08:10.020011 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:10.019949 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1086b45c9e8e98e78ef642018522560b24a9e63c7ce83bc58eea33de57d8f4"} err="failed to get container status \"bc1086b45c9e8e98e78ef642018522560b24a9e63c7ce83bc58eea33de57d8f4\": rpc error: code = NotFound desc = could not find container \"bc1086b45c9e8e98e78ef642018522560b24a9e63c7ce83bc58eea33de57d8f4\": container with ID starting with bc1086b45c9e8e98e78ef642018522560b24a9e63c7ce83bc58eea33de57d8f4 not found: ID does not exist" Nov 03 22:08:10.177033 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:10.176996 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58d66d58db-phhsh"] Nov 03 22:08:10.180025 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:10.180002 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-58d66d58db-phhsh"] Nov 03 22:08:10.625533 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:10.625454 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aacf527d-b30b-48cd-b5fd-be3eba9a2095" path="/var/lib/kubelet/pods/aacf527d-b30b-48cd-b5fd-be3eba9a2095/volumes" Nov 03 22:08:12.021629 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:08:12.021591 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:08:42.021471 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:08:42.021431 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:08:43.139744 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:43.139646 5084 generic.go:356] "Generic (PLEG): container finished" podID="60e01c32-cde7-4c5e-add4-ca8c17e82e34" containerID="891ba1ef7bacb3217c63e98ad5d5f6779dc848c55c2aecc39bae8c04069083f1" exitCode=0 Nov 03 22:08:43.139744 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:43.139730 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fqp5x" event={"ID":"60e01c32-cde7-4c5e-add4-ca8c17e82e34","Type":"ContainerDied","Data":"891ba1ef7bacb3217c63e98ad5d5f6779dc848c55c2aecc39bae8c04069083f1"} Nov 03 22:08:43.140307 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:43.139792 5084 scope.go:122] "RemoveContainer" containerID="6d8b523fd09bacad32ffe9c4d48457dbd1a2efd39af5487f0eb68a70e6ae2a13" Nov 03 22:08:43.140307 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:43.140179 5084 scope.go:122] "RemoveContainer" containerID="891ba1ef7bacb3217c63e98ad5d5f6779dc848c55c2aecc39bae8c04069083f1" Nov 03 22:08:43.140450 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:08:43.140428 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-fqp5x_openshift-insights(60e01c32-cde7-4c5e-add4-ca8c17e82e34)\"" pod="openshift-insights/insights-operator-7c4986d449-fqp5x" podUID="60e01c32-cde7-4c5e-add4-ca8c17e82e34" Nov 03 22:08:54.621462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:54.621423 5084 scope.go:122] "RemoveContainer" containerID="891ba1ef7bacb3217c63e98ad5d5f6779dc848c55c2aecc39bae8c04069083f1" Nov 03 22:08:55.192103 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:55.192045 5084 generic.go:356] "Generic (PLEG): container finished" podID="6884fe53-de40-4011-9f82-485876d56218" containerID="18dee5532db44e5169bfd67de9674ff8614ffcadb3f350ea2af54322c700a2a6" exitCode=0 Nov 03 22:08:55.192465 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:55.192117 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" event={"ID":"6884fe53-de40-4011-9f82-485876d56218","Type":"ContainerDied","Data":"18dee5532db44e5169bfd67de9674ff8614ffcadb3f350ea2af54322c700a2a6"} Nov 03 22:08:55.194203 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:55.194170 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-fqp5x" event={"ID":"60e01c32-cde7-4c5e-add4-ca8c17e82e34","Type":"ContainerStarted","Data":"31c864b8267a4aef45098cd59cebf0771494886ca00af1b983ccb12a6ba4602b"} Nov 03 22:08:55.671870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:55.671831 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4xsj8-t7xhr"] Nov 03 22:08:56.352763 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:56.352730 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" Nov 03 22:08:56.441896 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:56.441850 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6884fe53-de40-4011-9f82-485876d56218-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6884fe53-de40-4011-9f82-485876d56218-archives-path\") pod \"6884fe53-de40-4011-9f82-485876d56218\" (UID: \"6884fe53-de40-4011-9f82-485876d56218\") " Nov 03 22:08:56.441896 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:56.441898 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6884fe53-de40-4011-9f82-485876d56218-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6884fe53-de40-4011-9f82-485876d56218-trusted-ca-bundle\") pod \"6884fe53-de40-4011-9f82-485876d56218\" (UID: \"6884fe53-de40-4011-9f82-485876d56218\") " Nov 03 22:08:56.442181 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:56.441927 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6884fe53-de40-4011-9f82-485876d56218-kube-api-access-4jxsb\" (UniqueName: \"kubernetes.io/projected/6884fe53-de40-4011-9f82-485876d56218-kube-api-access-4jxsb\") pod \"6884fe53-de40-4011-9f82-485876d56218\" (UID: \"6884fe53-de40-4011-9f82-485876d56218\") " Nov 03 22:08:56.442181 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:56.441979 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6884fe53-de40-4011-9f82-485876d56218-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6884fe53-de40-4011-9f82-485876d56218-service-ca-bundle\") pod \"6884fe53-de40-4011-9f82-485876d56218\" (UID: \"6884fe53-de40-4011-9f82-485876d56218\") " Nov 03 22:08:56.442357 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:56.442331 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6884fe53-de40-4011-9f82-485876d56218-archives-path" pod "6884fe53-de40-4011-9f82-485876d56218" (UID: "6884fe53-de40-4011-9f82-485876d56218"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:08:56.442412 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:56.442376 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6884fe53-de40-4011-9f82-485876d56218-trusted-ca-bundle" pod "6884fe53-de40-4011-9f82-485876d56218" (UID: "6884fe53-de40-4011-9f82-485876d56218"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:56.442412 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:56.442405 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6884fe53-de40-4011-9f82-485876d56218-service-ca-bundle" pod "6884fe53-de40-4011-9f82-485876d56218" (UID: "6884fe53-de40-4011-9f82-485876d56218"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:08:56.444143 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:56.444124 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6884fe53-de40-4011-9f82-485876d56218-kube-api-access-4jxsb" pod "6884fe53-de40-4011-9f82-485876d56218" (UID: "6884fe53-de40-4011-9f82-485876d56218"). InnerVolumeSpecName "kube-api-access-4jxsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:08:56.543457 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:56.543409 5084 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6884fe53-de40-4011-9f82-485876d56218-archives-path\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:56.543457 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:56.543448 5084 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6884fe53-de40-4011-9f82-485876d56218-trusted-ca-bundle\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:56.543457 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:56.543462 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jxsb\" (UniqueName: \"kubernetes.io/projected/6884fe53-de40-4011-9f82-485876d56218-kube-api-access-4jxsb\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:56.543703 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:56.543476 5084 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6884fe53-de40-4011-9f82-485876d56218-service-ca-bundle\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:08:57.203237 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:57.203198 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" event={"ID":"6884fe53-de40-4011-9f82-485876d56218","Type":"ContainerDied","Data":"9e3ce94f023d943a6f320f6c400a8cc10ae56f3aa02ea953c522d9cf7ecbcc3c"} Nov 03 22:08:57.203644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:57.203250 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e3ce94f023d943a6f320f6c400a8cc10ae56f3aa02ea953c522d9cf7ecbcc3c" Nov 03 22:08:57.203644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:57.203213 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4xsj8-t7xhr" Nov 03 22:08:57.429017 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:57.428982 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4xsj8-t7xhr"] Nov 03 22:08:57.433005 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:57.432973 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-4xsj8-t7xhr"] Nov 03 22:08:58.626657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:08:58.626618 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6884fe53-de40-4011-9f82-485876d56218" path="/var/lib/kubelet/pods/6884fe53-de40-4011-9f82-485876d56218/volumes" Nov 03 22:09:12.021717 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:09:12.021676 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:09:42.022168 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:09:42.022082 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:10:12.021790 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:10:12.021754 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:10:42.020760 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:10:42.020718 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:11:12.025548 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:11:12.025500 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:11:30.228338 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.228300 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk"] Nov 03 22:11:30.228814 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.228646 5084 cpu_manager.go:414] "containerMap: removing container" podUID="aacf527d-b30b-48cd-b5fd-be3eba9a2095" containerName="registry" Nov 03 22:11:30.228814 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.228661 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aacf527d-b30b-48cd-b5fd-be3eba9a2095" containerName="registry" Nov 03 22:11:30.228814 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.228678 5084 cpu_manager.go:414] "containerMap: removing container" podUID="6884fe53-de40-4011-9f82-485876d56218" containerName="insights-gathering" Nov 03 22:11:30.228814 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.228683 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6884fe53-de40-4011-9f82-485876d56218" containerName="insights-gathering" Nov 03 22:11:30.228814 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.228689 5084 cpu_manager.go:414] "containerMap: removing container" podUID="19c22c7c-c7e9-4ef8-9e55-245393662f36" containerName="insights-gathering" Nov 03 22:11:30.228814 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.228695 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c22c7c-c7e9-4ef8-9e55-245393662f36" containerName="insights-gathering" Nov 03 22:11:30.228814 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.228716 5084 cpu_manager.go:414] "containerMap: removing container" podUID="7c0ac5b4-308e-4eca-8301-2c470cecdbef" containerName="registry" Nov 03 22:11:30.228814 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.228721 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c0ac5b4-308e-4eca-8301-2c470cecdbef" containerName="registry" Nov 03 22:11:30.228814 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.228791 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c22c7c-c7e9-4ef8-9e55-245393662f36" containerName="insights-gathering" Nov 03 22:11:30.228814 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.228804 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c0ac5b4-308e-4eca-8301-2c470cecdbef" containerName="registry" Nov 03 22:11:30.228814 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.228814 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="aacf527d-b30b-48cd-b5fd-be3eba9a2095" containerName="registry" Nov 03 22:11:30.229289 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.228823 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="6884fe53-de40-4011-9f82-485876d56218" containerName="insights-gathering" Nov 03 22:11:30.232146 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.232122 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" Nov 03 22:11:30.285215 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.285181 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk"] Nov 03 22:11:30.335356 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.335318 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 22:11:30.335545 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.335330 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 22:11:30.335545 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.335398 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-97844\"" Nov 03 22:11:30.358736 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.358700 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/808b5526-700b-49a0-8517-03eebaad0f55-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk\" (UID: \"808b5526-700b-49a0-8517-03eebaad0f55\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" Nov 03 22:11:30.358841 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.358746 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/808b5526-700b-49a0-8517-03eebaad0f55-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk\" (UID: \"808b5526-700b-49a0-8517-03eebaad0f55\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" Nov 03 22:11:30.358898 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.358863 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq6p8\" (UniqueName: \"kubernetes.io/projected/808b5526-700b-49a0-8517-03eebaad0f55-kube-api-access-xq6p8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk\" (UID: \"808b5526-700b-49a0-8517-03eebaad0f55\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" Nov 03 22:11:30.459672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.459626 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq6p8\" (UniqueName: \"kubernetes.io/projected/808b5526-700b-49a0-8517-03eebaad0f55-kube-api-access-xq6p8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk\" (UID: \"808b5526-700b-49a0-8517-03eebaad0f55\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" Nov 03 22:11:30.459858 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.459694 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/808b5526-700b-49a0-8517-03eebaad0f55-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk\" (UID: \"808b5526-700b-49a0-8517-03eebaad0f55\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" Nov 03 22:11:30.459858 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.459750 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/808b5526-700b-49a0-8517-03eebaad0f55-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk\" (UID: \"808b5526-700b-49a0-8517-03eebaad0f55\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" Nov 03 22:11:30.460108 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.460059 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/808b5526-700b-49a0-8517-03eebaad0f55-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk\" (UID: \"808b5526-700b-49a0-8517-03eebaad0f55\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" Nov 03 22:11:30.460108 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.460099 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/808b5526-700b-49a0-8517-03eebaad0f55-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk\" (UID: \"808b5526-700b-49a0-8517-03eebaad0f55\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" Nov 03 22:11:30.518681 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.518640 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq6p8\" (UniqueName: \"kubernetes.io/projected/808b5526-700b-49a0-8517-03eebaad0f55-kube-api-access-xq6p8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk\" (UID: \"808b5526-700b-49a0-8517-03eebaad0f55\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" Nov 03 22:11:30.548295 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.548259 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" Nov 03 22:11:30.922641 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:30.922562 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk"] Nov 03 22:11:30.956284 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:11:30.956245 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808b5526_700b_49a0_8517_03eebaad0f55.slice/crio-90d9b78ae1020b289f6434bac146a91e1eb233eb1237edef7fb37e1688327dc0 WatchSource:0}: Error finding container 90d9b78ae1020b289f6434bac146a91e1eb233eb1237edef7fb37e1688327dc0: Status 404 returned error can't find the container with id 90d9b78ae1020b289f6434bac146a91e1eb233eb1237edef7fb37e1688327dc0 Nov 03 22:11:31.628073 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.628035 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh"] Nov 03 22:11:31.632444 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.632417 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" Nov 03 22:11:31.684530 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.684488 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh"] Nov 03 22:11:31.734769 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.734732 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-ppsbk\"" Nov 03 22:11:31.735684 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.735661 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk"] Nov 03 22:11:31.739021 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.738999 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" Nov 03 22:11:31.770029 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.769991 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d478593a-5cf8-483c-b3b3-a3de12a14acc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh\" (UID: \"d478593a-5cf8-483c-b3b3-a3de12a14acc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" Nov 03 22:11:31.770242 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.770037 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdbq4\" (UniqueName: \"kubernetes.io/projected/d478593a-5cf8-483c-b3b3-a3de12a14acc-kube-api-access-bdbq4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh\" (UID: \"d478593a-5cf8-483c-b3b3-a3de12a14acc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" Nov 03 22:11:31.770242 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.770074 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d478593a-5cf8-483c-b3b3-a3de12a14acc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh\" (UID: \"d478593a-5cf8-483c-b3b3-a3de12a14acc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" Nov 03 22:11:31.778816 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.778784 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk"] Nov 03 22:11:31.792622 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.792590 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-cllp9\"" Nov 03 22:11:31.815337 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.815296 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" event={"ID":"808b5526-700b-49a0-8517-03eebaad0f55","Type":"ContainerStarted","Data":"90d9b78ae1020b289f6434bac146a91e1eb233eb1237edef7fb37e1688327dc0"} Nov 03 22:11:31.830029 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.829995 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75"] Nov 03 22:11:31.833483 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.833463 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75" Nov 03 22:11:31.871550 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.871515 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/431bd183-1d23-4665-84c1-7ee481e24f3b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk\" (UID: \"431bd183-1d23-4665-84c1-7ee481e24f3b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" Nov 03 22:11:31.871550 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.871555 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d478593a-5cf8-483c-b3b3-a3de12a14acc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh\" (UID: \"d478593a-5cf8-483c-b3b3-a3de12a14acc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" Nov 03 22:11:31.871767 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.871578 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdbq4\" (UniqueName: \"kubernetes.io/projected/d478593a-5cf8-483c-b3b3-a3de12a14acc-kube-api-access-bdbq4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh\" (UID: \"d478593a-5cf8-483c-b3b3-a3de12a14acc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" Nov 03 22:11:31.871767 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.871595 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d478593a-5cf8-483c-b3b3-a3de12a14acc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh\" (UID: \"d478593a-5cf8-483c-b3b3-a3de12a14acc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" Nov 03 22:11:31.871767 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.871723 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv5zv\" (UniqueName: \"kubernetes.io/projected/431bd183-1d23-4665-84c1-7ee481e24f3b-kube-api-access-zv5zv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk\" (UID: \"431bd183-1d23-4665-84c1-7ee481e24f3b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" Nov 03 22:11:31.871873 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.871781 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/431bd183-1d23-4665-84c1-7ee481e24f3b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk\" (UID: \"431bd183-1d23-4665-84c1-7ee481e24f3b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" Nov 03 22:11:31.871988 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.871971 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d478593a-5cf8-483c-b3b3-a3de12a14acc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh\" (UID: \"d478593a-5cf8-483c-b3b3-a3de12a14acc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" Nov 03 22:11:31.872030 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.871997 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d478593a-5cf8-483c-b3b3-a3de12a14acc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh\" (UID: \"d478593a-5cf8-483c-b3b3-a3de12a14acc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" Nov 03 22:11:31.886428 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.886343 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75"] Nov 03 22:11:31.892337 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.892306 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-r9hcn\"" Nov 03 22:11:31.924135 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.924098 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px"] Nov 03 22:11:31.927440 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.927414 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" Nov 03 22:11:31.930263 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.930237 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdbq4\" (UniqueName: \"kubernetes.io/projected/d478593a-5cf8-483c-b3b3-a3de12a14acc-kube-api-access-bdbq4\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh\" (UID: \"d478593a-5cf8-483c-b3b3-a3de12a14acc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" Nov 03 22:11:31.949395 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.949367 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" Nov 03 22:11:31.973031 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.972999 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aa085f8-fb14-47f1-b920-aef96c6707b3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75\" (UID: \"3aa085f8-fb14-47f1-b920-aef96c6707b3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75" Nov 03 22:11:31.973031 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.973041 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3aa085f8-fb14-47f1-b920-aef96c6707b3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75\" (UID: \"3aa085f8-fb14-47f1-b920-aef96c6707b3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75" Nov 03 22:11:31.973245 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.973151 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv5zv\" (UniqueName: \"kubernetes.io/projected/431bd183-1d23-4665-84c1-7ee481e24f3b-kube-api-access-zv5zv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk\" (UID: \"431bd183-1d23-4665-84c1-7ee481e24f3b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" Nov 03 22:11:31.973295 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.973265 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/431bd183-1d23-4665-84c1-7ee481e24f3b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk\" (UID: \"431bd183-1d23-4665-84c1-7ee481e24f3b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" Nov 03 22:11:31.973360 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.973332 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/431bd183-1d23-4665-84c1-7ee481e24f3b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk\" (UID: \"431bd183-1d23-4665-84c1-7ee481e24f3b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" Nov 03 22:11:31.973499 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.973389 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dnk4\" (UniqueName: \"kubernetes.io/projected/3aa085f8-fb14-47f1-b920-aef96c6707b3-kube-api-access-8dnk4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75\" (UID: \"3aa085f8-fb14-47f1-b920-aef96c6707b3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75" Nov 03 22:11:31.974438 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.973702 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/431bd183-1d23-4665-84c1-7ee481e24f3b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk\" (UID: \"431bd183-1d23-4665-84c1-7ee481e24f3b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" Nov 03 22:11:31.974438 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.973725 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/431bd183-1d23-4665-84c1-7ee481e24f3b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk\" (UID: \"431bd183-1d23-4665-84c1-7ee481e24f3b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" Nov 03 22:11:31.979886 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.979852 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px"] Nov 03 22:11:31.992670 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:31.992631 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-mcbd7\"" Nov 03 22:11:32.031736 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.031697 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv5zv\" (UniqueName: \"kubernetes.io/projected/431bd183-1d23-4665-84c1-7ee481e24f3b-kube-api-access-zv5zv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk\" (UID: \"431bd183-1d23-4665-84c1-7ee481e24f3b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" Nov 03 22:11:32.055322 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.055288 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" Nov 03 22:11:32.073830 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.073795 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dnk4\" (UniqueName: \"kubernetes.io/projected/3aa085f8-fb14-47f1-b920-aef96c6707b3-kube-api-access-8dnk4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75\" (UID: \"3aa085f8-fb14-47f1-b920-aef96c6707b3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75" Nov 03 22:11:32.073976 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.073848 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1209b357-ea25-4a54-91ac-2da8e81c7fdd-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px\" (UID: \"1209b357-ea25-4a54-91ac-2da8e81c7fdd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" Nov 03 22:11:32.073976 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.073869 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aa085f8-fb14-47f1-b920-aef96c6707b3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75\" (UID: \"3aa085f8-fb14-47f1-b920-aef96c6707b3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75" Nov 03 22:11:32.073976 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.073891 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3aa085f8-fb14-47f1-b920-aef96c6707b3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75\" (UID: \"3aa085f8-fb14-47f1-b920-aef96c6707b3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75" Nov 03 22:11:32.073976 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.073920 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1209b357-ea25-4a54-91ac-2da8e81c7fdd-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px\" (UID: \"1209b357-ea25-4a54-91ac-2da8e81c7fdd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" Nov 03 22:11:32.073976 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.073945 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbh8j\" (UniqueName: \"kubernetes.io/projected/1209b357-ea25-4a54-91ac-2da8e81c7fdd-kube-api-access-qbh8j\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px\" (UID: \"1209b357-ea25-4a54-91ac-2da8e81c7fdd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" Nov 03 22:11:32.074746 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.074716 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aa085f8-fb14-47f1-b920-aef96c6707b3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75\" (UID: \"3aa085f8-fb14-47f1-b920-aef96c6707b3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75" Nov 03 22:11:32.075045 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.074969 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3aa085f8-fb14-47f1-b920-aef96c6707b3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75\" (UID: \"3aa085f8-fb14-47f1-b920-aef96c6707b3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75" Nov 03 22:11:32.133965 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.133897 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dnk4\" (UniqueName: \"kubernetes.io/projected/3aa085f8-fb14-47f1-b920-aef96c6707b3-kube-api-access-8dnk4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75\" (UID: \"3aa085f8-fb14-47f1-b920-aef96c6707b3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75" Nov 03 22:11:32.149855 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.149759 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75" Nov 03 22:11:32.175409 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.175375 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1209b357-ea25-4a54-91ac-2da8e81c7fdd-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px\" (UID: \"1209b357-ea25-4a54-91ac-2da8e81c7fdd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" Nov 03 22:11:32.175572 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.175434 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1209b357-ea25-4a54-91ac-2da8e81c7fdd-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px\" (UID: \"1209b357-ea25-4a54-91ac-2da8e81c7fdd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" Nov 03 22:11:32.175572 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.175479 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbh8j\" (UniqueName: \"kubernetes.io/projected/1209b357-ea25-4a54-91ac-2da8e81c7fdd-kube-api-access-qbh8j\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px\" (UID: \"1209b357-ea25-4a54-91ac-2da8e81c7fdd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" Nov 03 22:11:32.176170 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.176148 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1209b357-ea25-4a54-91ac-2da8e81c7fdd-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px\" (UID: \"1209b357-ea25-4a54-91ac-2da8e81c7fdd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" Nov 03 22:11:32.176414 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.176398 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1209b357-ea25-4a54-91ac-2da8e81c7fdd-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px\" (UID: \"1209b357-ea25-4a54-91ac-2da8e81c7fdd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" Nov 03 22:11:32.234900 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.234863 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbh8j\" (UniqueName: \"kubernetes.io/projected/1209b357-ea25-4a54-91ac-2da8e81c7fdd-kube-api-access-qbh8j\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px\" (UID: \"1209b357-ea25-4a54-91ac-2da8e81c7fdd\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" Nov 03 22:11:32.250667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.250631 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" Nov 03 22:11:32.330516 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.330489 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh"] Nov 03 22:11:32.333415 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:11:32.333379 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd478593a_5cf8_483c_b3b3_a3de12a14acc.slice/crio-739359b71251e5c1bdf64c715aa397dfa3f0a8a899e5ff75d707112e5d2e6aad WatchSource:0}: Error finding container 739359b71251e5c1bdf64c715aa397dfa3f0a8a899e5ff75d707112e5d2e6aad: Status 404 returned error can't find the container with id 739359b71251e5c1bdf64c715aa397dfa3f0a8a899e5ff75d707112e5d2e6aad Nov 03 22:11:32.442693 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.442606 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk"] Nov 03 22:11:32.447109 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:11:32.447072 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431bd183_1d23_4665_84c1_7ee481e24f3b.slice/crio-8c1be663f10d8e596c46d7d2fb0a03a75f43135ba4286c77e29d9f9b54365d5c WatchSource:0}: Error finding container 8c1be663f10d8e596c46d7d2fb0a03a75f43135ba4286c77e29d9f9b54365d5c: Status 404 returned error can't find the container with id 8c1be663f10d8e596c46d7d2fb0a03a75f43135ba4286c77e29d9f9b54365d5c Nov 03 22:11:32.530922 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.530890 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75"] Nov 03 22:11:32.534461 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:11:32.534426 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa085f8_fb14_47f1_b920_aef96c6707b3.slice/crio-b3899c924db9001e5b0645f9cae964dd4afa3bf594986a4ee1147e0ebfd32475 WatchSource:0}: Error finding container b3899c924db9001e5b0645f9cae964dd4afa3bf594986a4ee1147e0ebfd32475: Status 404 returned error can't find the container with id b3899c924db9001e5b0645f9cae964dd4afa3bf594986a4ee1147e0ebfd32475 Nov 03 22:11:32.638844 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.638810 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px"] Nov 03 22:11:32.641669 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:11:32.641625 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1209b357_ea25_4a54_91ac_2da8e81c7fdd.slice/crio-b1651b13cc3119b1a11ed4bd903b0c6e95e53dae7e0efd6a97a5f1253120de75 WatchSource:0}: Error finding container b1651b13cc3119b1a11ed4bd903b0c6e95e53dae7e0efd6a97a5f1253120de75: Status 404 returned error can't find the container with id b1651b13cc3119b1a11ed4bd903b0c6e95e53dae7e0efd6a97a5f1253120de75 Nov 03 22:11:32.822899 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.822784 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" event={"ID":"431bd183-1d23-4665-84c1-7ee481e24f3b","Type":"ContainerStarted","Data":"8c1be663f10d8e596c46d7d2fb0a03a75f43135ba4286c77e29d9f9b54365d5c"} Nov 03 22:11:32.823966 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.823931 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" event={"ID":"1209b357-ea25-4a54-91ac-2da8e81c7fdd","Type":"ContainerStarted","Data":"b1651b13cc3119b1a11ed4bd903b0c6e95e53dae7e0efd6a97a5f1253120de75"} Nov 03 22:11:32.825435 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.825402 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75" event={"ID":"3aa085f8-fb14-47f1-b920-aef96c6707b3","Type":"ContainerStarted","Data":"b3899c924db9001e5b0645f9cae964dd4afa3bf594986a4ee1147e0ebfd32475"} Nov 03 22:11:32.826650 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:32.826618 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" event={"ID":"d478593a-5cf8-483c-b3b3-a3de12a14acc","Type":"ContainerStarted","Data":"739359b71251e5c1bdf64c715aa397dfa3f0a8a899e5ff75d707112e5d2e6aad"} Nov 03 22:11:33.690959 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.690919 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p"] Nov 03 22:11:33.695093 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.695043 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p" Nov 03 22:11:33.750106 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.747164 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p"] Nov 03 22:11:33.790281 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.790241 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8638335f-d656-4d5c-b04c-f3c688d6eb98-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p\" (UID: \"8638335f-d656-4d5c-b04c-f3c688d6eb98\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p" Nov 03 22:11:33.790473 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.790312 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8638335f-d656-4d5c-b04c-f3c688d6eb98-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p\" (UID: \"8638335f-d656-4d5c-b04c-f3c688d6eb98\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p" Nov 03 22:11:33.790473 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.790357 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lp89\" (UniqueName: \"kubernetes.io/projected/8638335f-d656-4d5c-b04c-f3c688d6eb98-kube-api-access-6lp89\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p\" (UID: \"8638335f-d656-4d5c-b04c-f3c688d6eb98\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p" Nov 03 22:11:33.797555 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.797521 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-ql597\"" Nov 03 22:11:33.804475 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.804439 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc"] Nov 03 22:11:33.809083 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.809044 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc" Nov 03 22:11:33.863341 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.863304 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-srt9c\"" Nov 03 22:11:33.864223 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.864188 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc"] Nov 03 22:11:33.892951 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.892909 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc\" (UID: \"d3fdc0a0-7096-44a5-b8b3-1bb30612143f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc" Nov 03 22:11:33.893187 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.893022 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8638335f-d656-4d5c-b04c-f3c688d6eb98-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p\" (UID: \"8638335f-d656-4d5c-b04c-f3c688d6eb98\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p" Nov 03 22:11:33.893187 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.893088 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8638335f-d656-4d5c-b04c-f3c688d6eb98-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p\" (UID: \"8638335f-d656-4d5c-b04c-f3c688d6eb98\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p" Nov 03 22:11:33.893187 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.893121 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc\" (UID: \"d3fdc0a0-7096-44a5-b8b3-1bb30612143f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc" Nov 03 22:11:33.893187 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.893156 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86tbw\" (UniqueName: \"kubernetes.io/projected/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-kube-api-access-86tbw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc\" (UID: \"d3fdc0a0-7096-44a5-b8b3-1bb30612143f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc" Nov 03 22:11:33.893482 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.893193 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lp89\" (UniqueName: \"kubernetes.io/projected/8638335f-d656-4d5c-b04c-f3c688d6eb98-kube-api-access-6lp89\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p\" (UID: \"8638335f-d656-4d5c-b04c-f3c688d6eb98\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p" Nov 03 22:11:33.894036 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.894005 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8638335f-d656-4d5c-b04c-f3c688d6eb98-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p\" (UID: \"8638335f-d656-4d5c-b04c-f3c688d6eb98\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p" Nov 03 22:11:33.894291 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.894271 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8638335f-d656-4d5c-b04c-f3c688d6eb98-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p\" (UID: \"8638335f-d656-4d5c-b04c-f3c688d6eb98\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p" Nov 03 22:11:33.964252 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.964176 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lp89\" (UniqueName: \"kubernetes.io/projected/8638335f-d656-4d5c-b04c-f3c688d6eb98-kube-api-access-6lp89\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p\" (UID: \"8638335f-d656-4d5c-b04c-f3c688d6eb98\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p" Nov 03 22:11:33.994401 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.994357 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc\" (UID: \"d3fdc0a0-7096-44a5-b8b3-1bb30612143f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc" Nov 03 22:11:33.994612 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.994418 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86tbw\" (UniqueName: \"kubernetes.io/projected/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-kube-api-access-86tbw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc\" (UID: \"d3fdc0a0-7096-44a5-b8b3-1bb30612143f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc" Nov 03 22:11:33.994612 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.994459 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc\" (UID: \"d3fdc0a0-7096-44a5-b8b3-1bb30612143f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc" Nov 03 22:11:33.994938 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.994914 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc\" (UID: \"d3fdc0a0-7096-44a5-b8b3-1bb30612143f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc" Nov 03 22:11:33.995193 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:33.995168 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc\" (UID: \"d3fdc0a0-7096-44a5-b8b3-1bb30612143f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc" Nov 03 22:11:34.016792 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:34.016753 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p" Nov 03 22:11:34.057957 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:34.057920 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86tbw\" (UniqueName: \"kubernetes.io/projected/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-kube-api-access-86tbw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc\" (UID: \"d3fdc0a0-7096-44a5-b8b3-1bb30612143f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc" Nov 03 22:11:34.129680 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:34.129640 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc" Nov 03 22:11:34.440799 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:34.440708 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p"] Nov 03 22:11:34.457584 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:11:34.457533 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8638335f_d656_4d5c_b04c_f3c688d6eb98.slice/crio-7efa77f3d6b089c49b316f7c493164a6bbcbfa15e381dbe8b714d477f97a1834 WatchSource:0}: Error finding container 7efa77f3d6b089c49b316f7c493164a6bbcbfa15e381dbe8b714d477f97a1834: Status 404 returned error can't find the container with id 7efa77f3d6b089c49b316f7c493164a6bbcbfa15e381dbe8b714d477f97a1834 Nov 03 22:11:34.552615 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:34.552588 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc"] Nov 03 22:11:34.844892 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:34.844718 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p" event={"ID":"8638335f-d656-4d5c-b04c-f3c688d6eb98","Type":"ContainerStarted","Data":"7efa77f3d6b089c49b316f7c493164a6bbcbfa15e381dbe8b714d477f97a1834"} Nov 03 22:11:34.848227 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:34.848105 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc" event={"ID":"d3fdc0a0-7096-44a5-b8b3-1bb30612143f","Type":"ContainerStarted","Data":"bb402bdaf6fdcef3d442e5704fe233f6f52ac1bd18095377387ecbbb77f38c26"} Nov 03 22:11:42.048353 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:11:42.048210 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:11:43.959144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:43.958982 5084 generic.go:356] "Generic (PLEG): container finished" podID="8638335f-d656-4d5c-b04c-f3c688d6eb98" containerID="16bf34120ee4d6c73e72b0df1c0eddc4c16d24d6b50869c08bcd337470ac397f" exitCode=0 Nov 03 22:11:43.959955 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:43.959919 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p" event={"ID":"8638335f-d656-4d5c-b04c-f3c688d6eb98","Type":"ContainerDied","Data":"16bf34120ee4d6c73e72b0df1c0eddc4c16d24d6b50869c08bcd337470ac397f"} Nov 03 22:11:43.962929 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:43.962205 5084 generic.go:356] "Generic (PLEG): container finished" podID="808b5526-700b-49a0-8517-03eebaad0f55" containerID="22165f5a9708640d57914b4bf8d79614a73a125b51e4de3e365bf1c3508994c3" exitCode=0 Nov 03 22:11:43.962929 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:43.962311 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" event={"ID":"808b5526-700b-49a0-8517-03eebaad0f55","Type":"ContainerDied","Data":"22165f5a9708640d57914b4bf8d79614a73a125b51e4de3e365bf1c3508994c3"} Nov 03 22:11:43.965896 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:43.965867 5084 generic.go:356] "Generic (PLEG): container finished" podID="d3fdc0a0-7096-44a5-b8b3-1bb30612143f" containerID="ee994b999f287d863aa88dcacb86b3b1f8ca9c81c7d2a997619f8051ebead3df" exitCode=0 Nov 03 22:11:43.966003 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:43.965920 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc" event={"ID":"d3fdc0a0-7096-44a5-b8b3-1bb30612143f","Type":"ContainerDied","Data":"ee994b999f287d863aa88dcacb86b3b1f8ca9c81c7d2a997619f8051ebead3df"} Nov 03 22:11:45.978680 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:45.978644 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" event={"ID":"431bd183-1d23-4665-84c1-7ee481e24f3b","Type":"ContainerStarted","Data":"06ab22994a74ee09386f8675f50fd2060d3005c15a3355cc537e852cb4bfdfeb"} Nov 03 22:11:45.980645 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:45.980618 5084 generic.go:356] "Generic (PLEG): container finished" podID="1209b357-ea25-4a54-91ac-2da8e81c7fdd" containerID="d0cbad50be05f6b056f1f5043c4226400890e18f002f37a5c27fa28d5651abe0" exitCode=0 Nov 03 22:11:45.980821 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:45.980691 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" event={"ID":"1209b357-ea25-4a54-91ac-2da8e81c7fdd","Type":"ContainerDied","Data":"d0cbad50be05f6b056f1f5043c4226400890e18f002f37a5c27fa28d5651abe0"} Nov 03 22:11:45.983552 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:45.983520 5084 generic.go:356] "Generic (PLEG): container finished" podID="d478593a-5cf8-483c-b3b3-a3de12a14acc" containerID="7fc2132f642135710b003ca63fb045201f685767d5aa9f7e650304a81aa00821" exitCode=0 Nov 03 22:11:45.983671 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:45.983651 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" event={"ID":"d478593a-5cf8-483c-b3b3-a3de12a14acc","Type":"ContainerDied","Data":"7fc2132f642135710b003ca63fb045201f685767d5aa9f7e650304a81aa00821"} Nov 03 22:11:45.986041 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:45.985982 5084 generic.go:356] "Generic (PLEG): container finished" podID="8638335f-d656-4d5c-b04c-f3c688d6eb98" containerID="51a57f5f7e7171e4abd313c8fc5942bea7b0ad81891a6cf6cd02caf1c1dae5e6" exitCode=0 Nov 03 22:11:45.986041 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:45.986028 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p" event={"ID":"8638335f-d656-4d5c-b04c-f3c688d6eb98","Type":"ContainerDied","Data":"51a57f5f7e7171e4abd313c8fc5942bea7b0ad81891a6cf6cd02caf1c1dae5e6"} Nov 03 22:11:46.993934 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:46.993887 5084 generic.go:356] "Generic (PLEG): container finished" podID="3aa085f8-fb14-47f1-b920-aef96c6707b3" containerID="acad60abb20b0de64e250f4b10b5527a236d0b40fd03ef158a4a1db1d7c1efef" exitCode=0 Nov 03 22:11:46.994427 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:46.994326 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75" event={"ID":"3aa085f8-fb14-47f1-b920-aef96c6707b3","Type":"ContainerDied","Data":"acad60abb20b0de64e250f4b10b5527a236d0b40fd03ef158a4a1db1d7c1efef"} Nov 03 22:11:46.996345 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:46.996298 5084 generic.go:356] "Generic (PLEG): container finished" podID="431bd183-1d23-4665-84c1-7ee481e24f3b" containerID="06ab22994a74ee09386f8675f50fd2060d3005c15a3355cc537e852cb4bfdfeb" exitCode=0 Nov 03 22:11:46.996345 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:46.996329 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" event={"ID":"431bd183-1d23-4665-84c1-7ee481e24f3b","Type":"ContainerDied","Data":"06ab22994a74ee09386f8675f50fd2060d3005c15a3355cc537e852cb4bfdfeb"} Nov 03 22:11:48.008594 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:48.004465 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" event={"ID":"1209b357-ea25-4a54-91ac-2da8e81c7fdd","Type":"ContainerStarted","Data":"cc8a6f01eaef92bb8d5ce2a9d8e9f3acc31ed79930e92bd0e5447f76526140db"} Nov 03 22:11:48.008594 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:48.007723 5084 generic.go:356] "Generic (PLEG): container finished" podID="d478593a-5cf8-483c-b3b3-a3de12a14acc" containerID="fe1204950c76533a3fdaa5f0e6e8cd56d8992df9abbccc4d1d5eb998f430dbc3" exitCode=0 Nov 03 22:11:48.008594 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:48.007858 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" event={"ID":"d478593a-5cf8-483c-b3b3-a3de12a14acc","Type":"ContainerDied","Data":"fe1204950c76533a3fdaa5f0e6e8cd56d8992df9abbccc4d1d5eb998f430dbc3"} Nov 03 22:11:48.011927 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:48.011237 5084 generic.go:356] "Generic (PLEG): container finished" podID="808b5526-700b-49a0-8517-03eebaad0f55" containerID="19538089edf5f1051b0b90c32dd04415ca3ea203c2c64288c2a5477a138bd506" exitCode=0 Nov 03 22:11:48.011927 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:48.011334 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" event={"ID":"808b5526-700b-49a0-8517-03eebaad0f55","Type":"ContainerDied","Data":"19538089edf5f1051b0b90c32dd04415ca3ea203c2c64288c2a5477a138bd506"} Nov 03 22:11:49.022517 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:49.022478 5084 generic.go:356] "Generic (PLEG): container finished" podID="1209b357-ea25-4a54-91ac-2da8e81c7fdd" containerID="cc8a6f01eaef92bb8d5ce2a9d8e9f3acc31ed79930e92bd0e5447f76526140db" exitCode=0 Nov 03 22:11:49.022999 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:49.022566 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" event={"ID":"1209b357-ea25-4a54-91ac-2da8e81c7fdd","Type":"ContainerDied","Data":"cc8a6f01eaef92bb8d5ce2a9d8e9f3acc31ed79930e92bd0e5447f76526140db"} Nov 03 22:11:49.026334 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:49.026304 5084 generic.go:356] "Generic (PLEG): container finished" podID="3aa085f8-fb14-47f1-b920-aef96c6707b3" containerID="61d455aa4779eeac658d84eb228786e42a8bc8e407bfa6d74c6c53c13bbf52c7" exitCode=0 Nov 03 22:11:49.026480 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:49.026448 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75" event={"ID":"3aa085f8-fb14-47f1-b920-aef96c6707b3","Type":"ContainerDied","Data":"61d455aa4779eeac658d84eb228786e42a8bc8e407bfa6d74c6c53c13bbf52c7"} Nov 03 22:11:49.029928 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:49.029885 5084 generic.go:356] "Generic (PLEG): container finished" podID="d3fdc0a0-7096-44a5-b8b3-1bb30612143f" containerID="fc96e5739049c59b6ff2e273909379e934469dbde06a5c0dcda79a0aee2762d0" exitCode=0 Nov 03 22:11:49.030353 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:49.030222 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc" event={"ID":"d3fdc0a0-7096-44a5-b8b3-1bb30612143f","Type":"ContainerDied","Data":"fc96e5739049c59b6ff2e273909379e934469dbde06a5c0dcda79a0aee2762d0"} Nov 03 22:11:49.042093 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:49.033639 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" event={"ID":"431bd183-1d23-4665-84c1-7ee481e24f3b","Type":"ContainerStarted","Data":"43d6ab82fa18b04b6bbd75a8161ed088d456920319c3cabcbee56becfa15cfec"} Nov 03 22:11:50.044053 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:50.044011 5084 generic.go:356] "Generic (PLEG): container finished" podID="431bd183-1d23-4665-84c1-7ee481e24f3b" containerID="43d6ab82fa18b04b6bbd75a8161ed088d456920319c3cabcbee56becfa15cfec" exitCode=0 Nov 03 22:11:50.044583 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:11:50.044081 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" event={"ID":"431bd183-1d23-4665-84c1-7ee481e24f3b","Type":"ContainerDied","Data":"43d6ab82fa18b04b6bbd75a8161ed088d456920319c3cabcbee56becfa15cfec"} Nov 03 22:12:00.144920 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:00.144838 5084 generic.go:356] "Generic (PLEG): container finished" podID="8638335f-d656-4d5c-b04c-f3c688d6eb98" containerID="e24b127cdda9be8f0536ccbdd31fab488e50c6a1e853050034236af028a87044" exitCode=0 Nov 03 22:12:00.145300 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:00.145002 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p" event={"ID":"8638335f-d656-4d5c-b04c-f3c688d6eb98","Type":"ContainerDied","Data":"e24b127cdda9be8f0536ccbdd31fab488e50c6a1e853050034236af028a87044"} Nov 03 22:12:00.151506 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:00.151466 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" event={"ID":"431bd183-1d23-4665-84c1-7ee481e24f3b","Type":"ContainerStarted","Data":"429562b149ca6150dfbbbe0fa7692117d900fde8507ce7e8d39234211fe37a86"} Nov 03 22:12:00.380910 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:00.380840 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" podStartSLOduration=4.237260688 podStartE2EDuration="29.380819174s" totalImagesPullingTime="25.143558486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:00.37885254 +0000 UTC m=+287.972884514" watchObservedRunningTime="2026-11-03 22:12:00.380819174 +0000 UTC m=+287.974851150" Nov 03 22:12:01.175713 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:01.174832 5084 generic.go:356] "Generic (PLEG): container finished" podID="431bd183-1d23-4665-84c1-7ee481e24f3b" containerID="429562b149ca6150dfbbbe0fa7692117d900fde8507ce7e8d39234211fe37a86" exitCode=0 Nov 03 22:12:01.175713 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:01.175643 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" event={"ID":"431bd183-1d23-4665-84c1-7ee481e24f3b","Type":"ContainerDied","Data":"429562b149ca6150dfbbbe0fa7692117d900fde8507ce7e8d39234211fe37a86"} Nov 03 22:12:01.394128 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:01.393975 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p" Nov 03 22:12:01.494640 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:01.494545 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8638335f-d656-4d5c-b04c-f3c688d6eb98-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8638335f-d656-4d5c-b04c-f3c688d6eb98-bundle\") pod \"8638335f-d656-4d5c-b04c-f3c688d6eb98\" (UID: \"8638335f-d656-4d5c-b04c-f3c688d6eb98\") " Nov 03 22:12:01.494837 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:01.494668 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8638335f-d656-4d5c-b04c-f3c688d6eb98-kube-api-access-6lp89\" (UniqueName: \"kubernetes.io/projected/8638335f-d656-4d5c-b04c-f3c688d6eb98-kube-api-access-6lp89\") pod \"8638335f-d656-4d5c-b04c-f3c688d6eb98\" (UID: \"8638335f-d656-4d5c-b04c-f3c688d6eb98\") " Nov 03 22:12:01.494837 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:01.494765 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8638335f-d656-4d5c-b04c-f3c688d6eb98-util\" (UniqueName: \"kubernetes.io/empty-dir/8638335f-d656-4d5c-b04c-f3c688d6eb98-util\") pod \"8638335f-d656-4d5c-b04c-f3c688d6eb98\" (UID: \"8638335f-d656-4d5c-b04c-f3c688d6eb98\") " Nov 03 22:12:01.495629 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:01.495580 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8638335f-d656-4d5c-b04c-f3c688d6eb98-bundle" pod "8638335f-d656-4d5c-b04c-f3c688d6eb98" (UID: "8638335f-d656-4d5c-b04c-f3c688d6eb98"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:01.501163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:01.501111 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8638335f-d656-4d5c-b04c-f3c688d6eb98-util" pod "8638335f-d656-4d5c-b04c-f3c688d6eb98" (UID: "8638335f-d656-4d5c-b04c-f3c688d6eb98"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:01.507884 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:01.507840 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8638335f-d656-4d5c-b04c-f3c688d6eb98-kube-api-access-6lp89" pod "8638335f-d656-4d5c-b04c-f3c688d6eb98" (UID: "8638335f-d656-4d5c-b04c-f3c688d6eb98"). InnerVolumeSpecName "kube-api-access-6lp89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:12:01.596360 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:01.596318 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lp89\" (UniqueName: \"kubernetes.io/projected/8638335f-d656-4d5c-b04c-f3c688d6eb98-kube-api-access-6lp89\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:01.596360 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:01.596355 5084 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8638335f-d656-4d5c-b04c-f3c688d6eb98-util\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:01.596360 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:01.596365 5084 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8638335f-d656-4d5c-b04c-f3c688d6eb98-bundle\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:02.187975 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:02.187900 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" event={"ID":"d478593a-5cf8-483c-b3b3-a3de12a14acc","Type":"ContainerStarted","Data":"96cca11a27d5e6de0b14d8a98fc2e9432aef2adab5df1da4ad6bef2ab78de2ef"} Nov 03 22:12:02.191516 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:02.191485 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p" Nov 03 22:12:02.191715 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:02.191568 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71m4l7p" event={"ID":"8638335f-d656-4d5c-b04c-f3c688d6eb98","Type":"ContainerDied","Data":"7efa77f3d6b089c49b316f7c493164a6bbcbfa15e381dbe8b714d477f97a1834"} Nov 03 22:12:02.191715 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:02.191621 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7efa77f3d6b089c49b316f7c493164a6bbcbfa15e381dbe8b714d477f97a1834" Nov 03 22:12:02.324462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:02.324398 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" podStartSLOduration=2.697499253 podStartE2EDuration="31.324376763s" totalImagesPullingTime="28.62687751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:02.323842218 +0000 UTC m=+289.917874197" watchObservedRunningTime="2026-11-03 22:12:02.324376763 +0000 UTC m=+289.918408736" Nov 03 22:12:02.389171 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:02.389129 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" Nov 03 22:12:02.505832 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:02.505790 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431bd183-1d23-4665-84c1-7ee481e24f3b-util\" (UniqueName: \"kubernetes.io/empty-dir/431bd183-1d23-4665-84c1-7ee481e24f3b-util\") pod \"431bd183-1d23-4665-84c1-7ee481e24f3b\" (UID: \"431bd183-1d23-4665-84c1-7ee481e24f3b\") " Nov 03 22:12:02.506024 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:02.505933 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431bd183-1d23-4665-84c1-7ee481e24f3b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/431bd183-1d23-4665-84c1-7ee481e24f3b-bundle\") pod \"431bd183-1d23-4665-84c1-7ee481e24f3b\" (UID: \"431bd183-1d23-4665-84c1-7ee481e24f3b\") " Nov 03 22:12:02.506024 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:02.505985 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/431bd183-1d23-4665-84c1-7ee481e24f3b-kube-api-access-zv5zv\" (UniqueName: \"kubernetes.io/projected/431bd183-1d23-4665-84c1-7ee481e24f3b-kube-api-access-zv5zv\") pod \"431bd183-1d23-4665-84c1-7ee481e24f3b\" (UID: \"431bd183-1d23-4665-84c1-7ee481e24f3b\") " Nov 03 22:12:02.507884 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:02.507832 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431bd183-1d23-4665-84c1-7ee481e24f3b-bundle" pod "431bd183-1d23-4665-84c1-7ee481e24f3b" (UID: "431bd183-1d23-4665-84c1-7ee481e24f3b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:02.515767 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:02.515721 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431bd183-1d23-4665-84c1-7ee481e24f3b-kube-api-access-zv5zv" pod "431bd183-1d23-4665-84c1-7ee481e24f3b" (UID: "431bd183-1d23-4665-84c1-7ee481e24f3b"). InnerVolumeSpecName "kube-api-access-zv5zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:12:02.522493 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:02.522404 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431bd183-1d23-4665-84c1-7ee481e24f3b-util" pod "431bd183-1d23-4665-84c1-7ee481e24f3b" (UID: "431bd183-1d23-4665-84c1-7ee481e24f3b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:02.607571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:02.607527 5084 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/431bd183-1d23-4665-84c1-7ee481e24f3b-bundle\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:02.607571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:02.607577 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv5zv\" (UniqueName: \"kubernetes.io/projected/431bd183-1d23-4665-84c1-7ee481e24f3b-kube-api-access-zv5zv\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:02.607831 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:02.607592 5084 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/431bd183-1d23-4665-84c1-7ee481e24f3b-util\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:03.201434 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:03.201395 5084 generic.go:356] "Generic (PLEG): container finished" podID="d478593a-5cf8-483c-b3b3-a3de12a14acc" containerID="96cca11a27d5e6de0b14d8a98fc2e9432aef2adab5df1da4ad6bef2ab78de2ef" exitCode=0 Nov 03 22:12:03.201892 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:03.201480 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" event={"ID":"d478593a-5cf8-483c-b3b3-a3de12a14acc","Type":"ContainerDied","Data":"96cca11a27d5e6de0b14d8a98fc2e9432aef2adab5df1da4ad6bef2ab78de2ef"} Nov 03 22:12:03.204544 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:03.204509 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" event={"ID":"808b5526-700b-49a0-8517-03eebaad0f55","Type":"ContainerStarted","Data":"60454d92cad0ad84c615ec232f8b8d2e4f5d34019604f788fa09369f73973eb1"} Nov 03 22:12:03.207688 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:03.207657 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" event={"ID":"431bd183-1d23-4665-84c1-7ee481e24f3b","Type":"ContainerDied","Data":"8c1be663f10d8e596c46d7d2fb0a03a75f43135ba4286c77e29d9f9b54365d5c"} Nov 03 22:12:03.207831 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:03.207708 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c1be663f10d8e596c46d7d2fb0a03a75f43135ba4286c77e29d9f9b54365d5c" Nov 03 22:12:03.207831 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:03.207726 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akpjdk" Nov 03 22:12:03.492961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:03.492908 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" podStartSLOduration=3.449115343 podStartE2EDuration="33.492891595s" totalImagesPullingTime="30.043776252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:11:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:03.492013097 +0000 UTC m=+291.086045072" watchObservedRunningTime="2026-11-03 22:12:03.492891595 +0000 UTC m=+291.086923566" Nov 03 22:12:04.214651 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:04.214555 5084 generic.go:356] "Generic (PLEG): container finished" podID="808b5526-700b-49a0-8517-03eebaad0f55" containerID="60454d92cad0ad84c615ec232f8b8d2e4f5d34019604f788fa09369f73973eb1" exitCode=0 Nov 03 22:12:04.215054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:04.214645 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" event={"ID":"808b5526-700b-49a0-8517-03eebaad0f55","Type":"ContainerDied","Data":"60454d92cad0ad84c615ec232f8b8d2e4f5d34019604f788fa09369f73973eb1"} Nov 03 22:12:04.216904 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:04.216874 5084 generic.go:356] "Generic (PLEG): container finished" podID="d3fdc0a0-7096-44a5-b8b3-1bb30612143f" containerID="6d429b81fbbeda4dca86bd3afe6c4ce7832a406c922da7d321b61c60cb2f8189" exitCode=0 Nov 03 22:12:04.217029 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:04.216957 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc" event={"ID":"d3fdc0a0-7096-44a5-b8b3-1bb30612143f","Type":"ContainerDied","Data":"6d429b81fbbeda4dca86bd3afe6c4ce7832a406c922da7d321b61c60cb2f8189"} Nov 03 22:12:04.219008 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:04.218982 5084 generic.go:356] "Generic (PLEG): container finished" podID="1209b357-ea25-4a54-91ac-2da8e81c7fdd" containerID="41a0af4dd2d8161cd8cff95479250b2c73b01ace2dd968e07698fd929d5be3a2" exitCode=0 Nov 03 22:12:04.219133 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:04.219049 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" event={"ID":"1209b357-ea25-4a54-91ac-2da8e81c7fdd","Type":"ContainerDied","Data":"41a0af4dd2d8161cd8cff95479250b2c73b01ace2dd968e07698fd929d5be3a2"} Nov 03 22:12:04.221116 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:04.221090 5084 generic.go:356] "Generic (PLEG): container finished" podID="3aa085f8-fb14-47f1-b920-aef96c6707b3" containerID="450c4ef1775dcd659fa0bb07780cd407ac08173180a86432c5ce1b02b72cb856" exitCode=0 Nov 03 22:12:04.221208 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:04.221169 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75" event={"ID":"3aa085f8-fb14-47f1-b920-aef96c6707b3","Type":"ContainerDied","Data":"450c4ef1775dcd659fa0bb07780cd407ac08173180a86432c5ce1b02b72cb856"} Nov 03 22:12:04.353946 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:04.353915 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" Nov 03 22:12:04.423817 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:04.423780 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d478593a-5cf8-483c-b3b3-a3de12a14acc-util\" (UniqueName: \"kubernetes.io/empty-dir/d478593a-5cf8-483c-b3b3-a3de12a14acc-util\") pod \"d478593a-5cf8-483c-b3b3-a3de12a14acc\" (UID: \"d478593a-5cf8-483c-b3b3-a3de12a14acc\") " Nov 03 22:12:04.424000 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:04.423833 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d478593a-5cf8-483c-b3b3-a3de12a14acc-kube-api-access-bdbq4\" (UniqueName: \"kubernetes.io/projected/d478593a-5cf8-483c-b3b3-a3de12a14acc-kube-api-access-bdbq4\") pod \"d478593a-5cf8-483c-b3b3-a3de12a14acc\" (UID: \"d478593a-5cf8-483c-b3b3-a3de12a14acc\") " Nov 03 22:12:04.424000 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:04.423867 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d478593a-5cf8-483c-b3b3-a3de12a14acc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d478593a-5cf8-483c-b3b3-a3de12a14acc-bundle\") pod \"d478593a-5cf8-483c-b3b3-a3de12a14acc\" (UID: \"d478593a-5cf8-483c-b3b3-a3de12a14acc\") " Nov 03 22:12:04.424460 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:04.424426 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d478593a-5cf8-483c-b3b3-a3de12a14acc-bundle" pod "d478593a-5cf8-483c-b3b3-a3de12a14acc" (UID: "d478593a-5cf8-483c-b3b3-a3de12a14acc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:04.426198 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:04.426160 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d478593a-5cf8-483c-b3b3-a3de12a14acc-kube-api-access-bdbq4" pod "d478593a-5cf8-483c-b3b3-a3de12a14acc" (UID: "d478593a-5cf8-483c-b3b3-a3de12a14acc"). InnerVolumeSpecName "kube-api-access-bdbq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:12:04.428132 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:04.428107 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d478593a-5cf8-483c-b3b3-a3de12a14acc-util" pod "d478593a-5cf8-483c-b3b3-a3de12a14acc" (UID: "d478593a-5cf8-483c-b3b3-a3de12a14acc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:04.524907 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:04.524873 5084 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d478593a-5cf8-483c-b3b3-a3de12a14acc-bundle\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:04.524907 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:04.524907 5084 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d478593a-5cf8-483c-b3b3-a3de12a14acc-util\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:04.525131 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:04.524917 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdbq4\" (UniqueName: \"kubernetes.io/projected/d478593a-5cf8-483c-b3b3-a3de12a14acc-kube-api-access-bdbq4\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:05.227374 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.227340 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" Nov 03 22:12:05.227374 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.227352 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7qbblh" event={"ID":"d478593a-5cf8-483c-b3b3-a3de12a14acc","Type":"ContainerDied","Data":"739359b71251e5c1bdf64c715aa397dfa3f0a8a899e5ff75d707112e5d2e6aad"} Nov 03 22:12:05.227918 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.227418 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="739359b71251e5c1bdf64c715aa397dfa3f0a8a899e5ff75d707112e5d2e6aad" Nov 03 22:12:05.381970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.381929 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc" Nov 03 22:12:05.428190 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.428160 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" Nov 03 22:12:05.430891 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.430865 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" Nov 03 22:12:05.431232 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.431210 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-kube-api-access-86tbw\" (UniqueName: \"kubernetes.io/projected/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-kube-api-access-86tbw\") pod \"d3fdc0a0-7096-44a5-b8b3-1bb30612143f\" (UID: \"d3fdc0a0-7096-44a5-b8b3-1bb30612143f\") " Nov 03 22:12:05.431286 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.431276 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-util\" (UniqueName: \"kubernetes.io/empty-dir/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-util\") pod \"d3fdc0a0-7096-44a5-b8b3-1bb30612143f\" (UID: \"d3fdc0a0-7096-44a5-b8b3-1bb30612143f\") " Nov 03 22:12:05.431335 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.431303 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-bundle\") pod \"d3fdc0a0-7096-44a5-b8b3-1bb30612143f\" (UID: \"d3fdc0a0-7096-44a5-b8b3-1bb30612143f\") " Nov 03 22:12:05.431735 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.431693 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-bundle" pod "d3fdc0a0-7096-44a5-b8b3-1bb30612143f" (UID: "d3fdc0a0-7096-44a5-b8b3-1bb30612143f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:05.433481 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.433454 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-kube-api-access-86tbw" pod "d3fdc0a0-7096-44a5-b8b3-1bb30612143f" (UID: "d3fdc0a0-7096-44a5-b8b3-1bb30612143f"). InnerVolumeSpecName "kube-api-access-86tbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:12:05.435255 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.435222 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-util" pod "d3fdc0a0-7096-44a5-b8b3-1bb30612143f" (UID: "d3fdc0a0-7096-44a5-b8b3-1bb30612143f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:05.444328 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.444296 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75" Nov 03 22:12:05.532383 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.532339 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/808b5526-700b-49a0-8517-03eebaad0f55-bundle\" (UniqueName: \"kubernetes.io/empty-dir/808b5526-700b-49a0-8517-03eebaad0f55-bundle\") pod \"808b5526-700b-49a0-8517-03eebaad0f55\" (UID: \"808b5526-700b-49a0-8517-03eebaad0f55\") " Nov 03 22:12:05.532576 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.532405 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1209b357-ea25-4a54-91ac-2da8e81c7fdd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1209b357-ea25-4a54-91ac-2da8e81c7fdd-bundle\") pod \"1209b357-ea25-4a54-91ac-2da8e81c7fdd\" (UID: \"1209b357-ea25-4a54-91ac-2da8e81c7fdd\") " Nov 03 22:12:05.532576 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.532448 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/808b5526-700b-49a0-8517-03eebaad0f55-kube-api-access-xq6p8\" (UniqueName: \"kubernetes.io/projected/808b5526-700b-49a0-8517-03eebaad0f55-kube-api-access-xq6p8\") pod \"808b5526-700b-49a0-8517-03eebaad0f55\" (UID: \"808b5526-700b-49a0-8517-03eebaad0f55\") " Nov 03 22:12:05.532576 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.532487 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa085f8-fb14-47f1-b920-aef96c6707b3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aa085f8-fb14-47f1-b920-aef96c6707b3-bundle\") pod \"3aa085f8-fb14-47f1-b920-aef96c6707b3\" (UID: \"3aa085f8-fb14-47f1-b920-aef96c6707b3\") " Nov 03 22:12:05.532576 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.532520 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa085f8-fb14-47f1-b920-aef96c6707b3-kube-api-access-8dnk4\" (UniqueName: \"kubernetes.io/projected/3aa085f8-fb14-47f1-b920-aef96c6707b3-kube-api-access-8dnk4\") pod \"3aa085f8-fb14-47f1-b920-aef96c6707b3\" (UID: \"3aa085f8-fb14-47f1-b920-aef96c6707b3\") " Nov 03 22:12:05.532576 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.532565 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1209b357-ea25-4a54-91ac-2da8e81c7fdd-util\" (UniqueName: \"kubernetes.io/empty-dir/1209b357-ea25-4a54-91ac-2da8e81c7fdd-util\") pod \"1209b357-ea25-4a54-91ac-2da8e81c7fdd\" (UID: \"1209b357-ea25-4a54-91ac-2da8e81c7fdd\") " Nov 03 22:12:05.532835 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.532617 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1209b357-ea25-4a54-91ac-2da8e81c7fdd-kube-api-access-qbh8j\" (UniqueName: \"kubernetes.io/projected/1209b357-ea25-4a54-91ac-2da8e81c7fdd-kube-api-access-qbh8j\") pod \"1209b357-ea25-4a54-91ac-2da8e81c7fdd\" (UID: \"1209b357-ea25-4a54-91ac-2da8e81c7fdd\") " Nov 03 22:12:05.532835 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.532653 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/808b5526-700b-49a0-8517-03eebaad0f55-util\" (UniqueName: \"kubernetes.io/empty-dir/808b5526-700b-49a0-8517-03eebaad0f55-util\") pod \"808b5526-700b-49a0-8517-03eebaad0f55\" (UID: \"808b5526-700b-49a0-8517-03eebaad0f55\") " Nov 03 22:12:05.532835 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.532686 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa085f8-fb14-47f1-b920-aef96c6707b3-util\" (UniqueName: \"kubernetes.io/empty-dir/3aa085f8-fb14-47f1-b920-aef96c6707b3-util\") pod \"3aa085f8-fb14-47f1-b920-aef96c6707b3\" (UID: \"3aa085f8-fb14-47f1-b920-aef96c6707b3\") " Nov 03 22:12:05.532981 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.532915 5084 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-bundle\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:05.532981 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.532935 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86tbw\" (UniqueName: \"kubernetes.io/projected/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-kube-api-access-86tbw\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:05.532981 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.532948 5084 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3fdc0a0-7096-44a5-b8b3-1bb30612143f-util\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:05.533153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.533004 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa085f8-fb14-47f1-b920-aef96c6707b3-bundle" pod "3aa085f8-fb14-47f1-b920-aef96c6707b3" (UID: "3aa085f8-fb14-47f1-b920-aef96c6707b3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:05.533153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.533058 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1209b357-ea25-4a54-91ac-2da8e81c7fdd-bundle" pod "1209b357-ea25-4a54-91ac-2da8e81c7fdd" (UID: "1209b357-ea25-4a54-91ac-2da8e81c7fdd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:05.533644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.533612 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808b5526-700b-49a0-8517-03eebaad0f55-bundle" pod "808b5526-700b-49a0-8517-03eebaad0f55" (UID: "808b5526-700b-49a0-8517-03eebaad0f55"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:05.535279 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.535245 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808b5526-700b-49a0-8517-03eebaad0f55-kube-api-access-xq6p8" pod "808b5526-700b-49a0-8517-03eebaad0f55" (UID: "808b5526-700b-49a0-8517-03eebaad0f55"). InnerVolumeSpecName "kube-api-access-xq6p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:12:05.535677 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.535644 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa085f8-fb14-47f1-b920-aef96c6707b3-kube-api-access-8dnk4" pod "3aa085f8-fb14-47f1-b920-aef96c6707b3" (UID: "3aa085f8-fb14-47f1-b920-aef96c6707b3"). InnerVolumeSpecName "kube-api-access-8dnk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:12:05.535788 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.535749 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1209b357-ea25-4a54-91ac-2da8e81c7fdd-kube-api-access-qbh8j" pod "1209b357-ea25-4a54-91ac-2da8e81c7fdd" (UID: "1209b357-ea25-4a54-91ac-2da8e81c7fdd"). InnerVolumeSpecName "kube-api-access-qbh8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:12:05.539189 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.539155 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808b5526-700b-49a0-8517-03eebaad0f55-util" pod "808b5526-700b-49a0-8517-03eebaad0f55" (UID: "808b5526-700b-49a0-8517-03eebaad0f55"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:05.540221 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.540195 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa085f8-fb14-47f1-b920-aef96c6707b3-util" pod "3aa085f8-fb14-47f1-b920-aef96c6707b3" (UID: "3aa085f8-fb14-47f1-b920-aef96c6707b3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:05.540519 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.540487 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1209b357-ea25-4a54-91ac-2da8e81c7fdd-util" pod "1209b357-ea25-4a54-91ac-2da8e81c7fdd" (UID: "1209b357-ea25-4a54-91ac-2da8e81c7fdd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:12:05.633720 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.633685 5084 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1209b357-ea25-4a54-91ac-2da8e81c7fdd-bundle\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:05.633720 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.633719 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq6p8\" (UniqueName: \"kubernetes.io/projected/808b5526-700b-49a0-8517-03eebaad0f55-kube-api-access-xq6p8\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:05.633937 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.633735 5084 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aa085f8-fb14-47f1-b920-aef96c6707b3-bundle\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:05.633937 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.633747 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dnk4\" (UniqueName: \"kubernetes.io/projected/3aa085f8-fb14-47f1-b920-aef96c6707b3-kube-api-access-8dnk4\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:05.633937 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.633759 5084 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1209b357-ea25-4a54-91ac-2da8e81c7fdd-util\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:05.633937 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.633771 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbh8j\" (UniqueName: \"kubernetes.io/projected/1209b357-ea25-4a54-91ac-2da8e81c7fdd-kube-api-access-qbh8j\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:05.633937 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.633782 5084 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/808b5526-700b-49a0-8517-03eebaad0f55-util\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:05.633937 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.633794 5084 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3aa085f8-fb14-47f1-b920-aef96c6707b3-util\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:05.633937 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:05.633805 5084 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/808b5526-700b-49a0-8517-03eebaad0f55-bundle\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:12:06.234206 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.234170 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" Nov 03 22:12:06.234640 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.234173 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cw7pvk" event={"ID":"808b5526-700b-49a0-8517-03eebaad0f55","Type":"ContainerDied","Data":"90d9b78ae1020b289f6434bac146a91e1eb233eb1237edef7fb37e1688327dc0"} Nov 03 22:12:06.234640 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.234306 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90d9b78ae1020b289f6434bac146a91e1eb233eb1237edef7fb37e1688327dc0" Nov 03 22:12:06.236251 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.236227 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc" Nov 03 22:12:06.236251 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.236238 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cqbrc" event={"ID":"d3fdc0a0-7096-44a5-b8b3-1bb30612143f","Type":"ContainerDied","Data":"bb402bdaf6fdcef3d442e5704fe233f6f52ac1bd18095377387ecbbb77f38c26"} Nov 03 22:12:06.236416 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.236289 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb402bdaf6fdcef3d442e5704fe233f6f52ac1bd18095377387ecbbb77f38c26" Nov 03 22:12:06.238286 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.238263 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" event={"ID":"1209b357-ea25-4a54-91ac-2da8e81c7fdd","Type":"ContainerDied","Data":"b1651b13cc3119b1a11ed4bd903b0c6e95e53dae7e0efd6a97a5f1253120de75"} Nov 03 22:12:06.238414 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.238305 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1651b13cc3119b1a11ed4bd903b0c6e95e53dae7e0efd6a97a5f1253120de75" Nov 03 22:12:06.238414 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.238272 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbc4px" Nov 03 22:12:06.240247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.240225 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75" event={"ID":"3aa085f8-fb14-47f1-b920-aef96c6707b3","Type":"ContainerDied","Data":"b3899c924db9001e5b0645f9cae964dd4afa3bf594986a4ee1147e0ebfd32475"} Nov 03 22:12:06.240349 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.240258 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3899c924db9001e5b0645f9cae964dd4afa3bf594986a4ee1147e0ebfd32475" Nov 03 22:12:06.240349 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.240260 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6l6h75" Nov 03 22:12:06.280798 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.280763 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5"] Nov 03 22:12:06.281107 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281094 5084 cpu_manager.go:414] "containerMap: removing container" podUID="8638335f-d656-4d5c-b04c-f3c688d6eb98" containerName="pull" Nov 03 22:12:06.281160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281108 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8638335f-d656-4d5c-b04c-f3c688d6eb98" containerName="pull" Nov 03 22:12:06.281160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281119 5084 cpu_manager.go:414] "containerMap: removing container" podUID="d478593a-5cf8-483c-b3b3-a3de12a14acc" containerName="extract" Nov 03 22:12:06.281160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281124 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d478593a-5cf8-483c-b3b3-a3de12a14acc" containerName="extract" Nov 03 22:12:06.281160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281134 5084 cpu_manager.go:414] "containerMap: removing container" podUID="8638335f-d656-4d5c-b04c-f3c688d6eb98" containerName="util" Nov 03 22:12:06.281160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281139 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8638335f-d656-4d5c-b04c-f3c688d6eb98" containerName="util" Nov 03 22:12:06.281160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281149 5084 cpu_manager.go:414] "containerMap: removing container" podUID="808b5526-700b-49a0-8517-03eebaad0f55" containerName="extract" Nov 03 22:12:06.281160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281154 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="808b5526-700b-49a0-8517-03eebaad0f55" containerName="extract" Nov 03 22:12:06.281160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281162 5084 cpu_manager.go:414] "containerMap: removing container" podUID="1209b357-ea25-4a54-91ac-2da8e81c7fdd" containerName="pull" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281169 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1209b357-ea25-4a54-91ac-2da8e81c7fdd" containerName="pull" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281175 5084 cpu_manager.go:414] "containerMap: removing container" podUID="3aa085f8-fb14-47f1-b920-aef96c6707b3" containerName="pull" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281180 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa085f8-fb14-47f1-b920-aef96c6707b3" containerName="pull" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281187 5084 cpu_manager.go:414] "containerMap: removing container" podUID="431bd183-1d23-4665-84c1-7ee481e24f3b" containerName="util" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281191 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="431bd183-1d23-4665-84c1-7ee481e24f3b" containerName="util" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281197 5084 cpu_manager.go:414] "containerMap: removing container" podUID="3aa085f8-fb14-47f1-b920-aef96c6707b3" containerName="util" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281201 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa085f8-fb14-47f1-b920-aef96c6707b3" containerName="util" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281207 5084 cpu_manager.go:414] "containerMap: removing container" podUID="d3fdc0a0-7096-44a5-b8b3-1bb30612143f" containerName="extract" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281212 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3fdc0a0-7096-44a5-b8b3-1bb30612143f" containerName="extract" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281224 5084 cpu_manager.go:414] "containerMap: removing container" podUID="d3fdc0a0-7096-44a5-b8b3-1bb30612143f" containerName="util" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281229 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3fdc0a0-7096-44a5-b8b3-1bb30612143f" containerName="util" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281235 5084 cpu_manager.go:414] "containerMap: removing container" podUID="1209b357-ea25-4a54-91ac-2da8e81c7fdd" containerName="util" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281239 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1209b357-ea25-4a54-91ac-2da8e81c7fdd" containerName="util" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281245 5084 cpu_manager.go:414] "containerMap: removing container" podUID="808b5526-700b-49a0-8517-03eebaad0f55" containerName="pull" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281248 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="808b5526-700b-49a0-8517-03eebaad0f55" containerName="pull" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281254 5084 cpu_manager.go:414] "containerMap: removing container" podUID="d478593a-5cf8-483c-b3b3-a3de12a14acc" containerName="pull" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281259 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d478593a-5cf8-483c-b3b3-a3de12a14acc" containerName="pull" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281265 5084 cpu_manager.go:414] "containerMap: removing container" podUID="431bd183-1d23-4665-84c1-7ee481e24f3b" containerName="extract" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281269 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="431bd183-1d23-4665-84c1-7ee481e24f3b" containerName="extract" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281276 5084 cpu_manager.go:414] "containerMap: removing container" podUID="1209b357-ea25-4a54-91ac-2da8e81c7fdd" containerName="extract" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281281 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1209b357-ea25-4a54-91ac-2da8e81c7fdd" containerName="extract" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281290 5084 cpu_manager.go:414] "containerMap: removing container" podUID="431bd183-1d23-4665-84c1-7ee481e24f3b" containerName="pull" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281295 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="431bd183-1d23-4665-84c1-7ee481e24f3b" containerName="pull" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281302 5084 cpu_manager.go:414] "containerMap: removing container" podUID="8638335f-d656-4d5c-b04c-f3c688d6eb98" containerName="extract" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281307 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8638335f-d656-4d5c-b04c-f3c688d6eb98" containerName="extract" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281316 5084 cpu_manager.go:414] "containerMap: removing container" podUID="d3fdc0a0-7096-44a5-b8b3-1bb30612143f" containerName="pull" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281320 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3fdc0a0-7096-44a5-b8b3-1bb30612143f" containerName="pull" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281330 5084 cpu_manager.go:414] "containerMap: removing container" podUID="d478593a-5cf8-483c-b3b3-a3de12a14acc" containerName="util" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281335 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d478593a-5cf8-483c-b3b3-a3de12a14acc" containerName="util" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281341 5084 cpu_manager.go:414] "containerMap: removing container" podUID="3aa085f8-fb14-47f1-b920-aef96c6707b3" containerName="extract" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281346 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa085f8-fb14-47f1-b920-aef96c6707b3" containerName="extract" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281353 5084 cpu_manager.go:414] "containerMap: removing container" podUID="808b5526-700b-49a0-8517-03eebaad0f55" containerName="util" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281357 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="808b5526-700b-49a0-8517-03eebaad0f55" containerName="util" Nov 03 22:12:06.281388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281400 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="d478593a-5cf8-483c-b3b3-a3de12a14acc" containerName="extract" Nov 03 22:12:06.282386 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281408 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="808b5526-700b-49a0-8517-03eebaad0f55" containerName="extract" Nov 03 22:12:06.282386 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281415 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3fdc0a0-7096-44a5-b8b3-1bb30612143f" containerName="extract" Nov 03 22:12:06.282386 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281421 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="431bd183-1d23-4665-84c1-7ee481e24f3b" containerName="extract" Nov 03 22:12:06.282386 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281427 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="1209b357-ea25-4a54-91ac-2da8e81c7fdd" containerName="extract" Nov 03 22:12:06.282386 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281432 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="8638335f-d656-4d5c-b04c-f3c688d6eb98" containerName="extract" Nov 03 22:12:06.282386 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.281440 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa085f8-fb14-47f1-b920-aef96c6707b3" containerName="extract" Nov 03 22:12:06.285905 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.285883 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.339760 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.339719 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b3865bd-0448-4c26-b256-4a1c1608a549-manifests\") pod \"opendatahub-operator-controller-manager-55b449f4f4-qtdf5\" (UID: \"7b3865bd-0448-4c26-b256-4a1c1608a549\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.339960 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.339767 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgz7s\" (UniqueName: \"kubernetes.io/projected/7b3865bd-0448-4c26-b256-4a1c1608a549-kube-api-access-hgz7s\") pod \"opendatahub-operator-controller-manager-55b449f4f4-qtdf5\" (UID: \"7b3865bd-0448-4c26-b256-4a1c1608a549\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.339960 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.339845 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b3865bd-0448-4c26-b256-4a1c1608a549-tmp\") pod \"opendatahub-operator-controller-manager-55b449f4f4-qtdf5\" (UID: \"7b3865bd-0448-4c26-b256-4a1c1608a549\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.339960 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.339914 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b3865bd-0448-4c26-b256-4a1c1608a549-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-qtdf5\" (UID: \"7b3865bd-0448-4c26-b256-4a1c1608a549\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.340170 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.339993 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b3865bd-0448-4c26-b256-4a1c1608a549-metrics-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-qtdf5\" (UID: \"7b3865bd-0448-4c26-b256-4a1c1608a549\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.340170 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.340040 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b3865bd-0448-4c26-b256-4a1c1608a549-webhook-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-qtdf5\" (UID: \"7b3865bd-0448-4c26-b256-4a1c1608a549\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.340984 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.340956 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 22:12:06.341096 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.340956 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 22:12:06.341096 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.341088 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 22:12:06.341214 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.341180 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 22:12:06.341214 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.341200 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-n7ktg\"" Nov 03 22:12:06.341285 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.341233 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 22:12:06.385587 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.385550 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5"] Nov 03 22:12:06.441221 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.441170 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b3865bd-0448-4c26-b256-4a1c1608a549-manifests\") pod \"opendatahub-operator-controller-manager-55b449f4f4-qtdf5\" (UID: \"7b3865bd-0448-4c26-b256-4a1c1608a549\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.441221 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.441224 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgz7s\" (UniqueName: \"kubernetes.io/projected/7b3865bd-0448-4c26-b256-4a1c1608a549-kube-api-access-hgz7s\") pod \"opendatahub-operator-controller-manager-55b449f4f4-qtdf5\" (UID: \"7b3865bd-0448-4c26-b256-4a1c1608a549\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.441426 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.441244 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b3865bd-0448-4c26-b256-4a1c1608a549-tmp\") pod \"opendatahub-operator-controller-manager-55b449f4f4-qtdf5\" (UID: \"7b3865bd-0448-4c26-b256-4a1c1608a549\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.441426 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.441266 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b3865bd-0448-4c26-b256-4a1c1608a549-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-qtdf5\" (UID: \"7b3865bd-0448-4c26-b256-4a1c1608a549\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.441426 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.441297 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b3865bd-0448-4c26-b256-4a1c1608a549-metrics-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-qtdf5\" (UID: \"7b3865bd-0448-4c26-b256-4a1c1608a549\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.441426 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.441321 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b3865bd-0448-4c26-b256-4a1c1608a549-webhook-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-qtdf5\" (UID: \"7b3865bd-0448-4c26-b256-4a1c1608a549\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.441713 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.441688 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b3865bd-0448-4c26-b256-4a1c1608a549-manifests\") pod \"opendatahub-operator-controller-manager-55b449f4f4-qtdf5\" (UID: \"7b3865bd-0448-4c26-b256-4a1c1608a549\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.443845 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.443816 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b3865bd-0448-4c26-b256-4a1c1608a549-webhook-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-qtdf5\" (UID: \"7b3865bd-0448-4c26-b256-4a1c1608a549\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.444010 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.443933 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b3865bd-0448-4c26-b256-4a1c1608a549-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-qtdf5\" (UID: \"7b3865bd-0448-4c26-b256-4a1c1608a549\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.444010 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.443987 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b3865bd-0448-4c26-b256-4a1c1608a549-tmp\") pod \"opendatahub-operator-controller-manager-55b449f4f4-qtdf5\" (UID: \"7b3865bd-0448-4c26-b256-4a1c1608a549\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.444143 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.444128 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b3865bd-0448-4c26-b256-4a1c1608a549-metrics-cert\") pod \"opendatahub-operator-controller-manager-55b449f4f4-qtdf5\" (UID: \"7b3865bd-0448-4c26-b256-4a1c1608a549\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.499914 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.499828 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgz7s\" (UniqueName: \"kubernetes.io/projected/7b3865bd-0448-4c26-b256-4a1c1608a549-kube-api-access-hgz7s\") pod \"opendatahub-operator-controller-manager-55b449f4f4-qtdf5\" (UID: \"7b3865bd-0448-4c26-b256-4a1c1608a549\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:06.601208 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:06.601169 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:07.101883 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:07.101850 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5"] Nov 03 22:12:07.104355 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:12:07.104322 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3865bd_0448_4c26_b256_4a1c1608a549.slice/crio-edc2963afe152964cab1c15c330643ecc44e5fb967bd45380b47b28c230088ab WatchSource:0}: Error finding container edc2963afe152964cab1c15c330643ecc44e5fb967bd45380b47b28c230088ab: Status 404 returned error can't find the container with id edc2963afe152964cab1c15c330643ecc44e5fb967bd45380b47b28c230088ab Nov 03 22:12:07.247058 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:07.247017 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" event={"ID":"7b3865bd-0448-4c26-b256-4a1c1608a549","Type":"ContainerStarted","Data":"edc2963afe152964cab1c15c330643ecc44e5fb967bd45380b47b28c230088ab"} Nov 03 22:12:11.270231 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:11.270188 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" event={"ID":"7b3865bd-0448-4c26-b256-4a1c1608a549","Type":"ContainerStarted","Data":"90088b9fecd6d774accc6ab233ac62eb98061694e9f6ccf18b93a2bc68fa4387"} Nov 03 22:12:12.023989 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:12:12.023947 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:12:12.276368 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:12.276273 5084 generic.go:356] "Generic (PLEG): container finished" podID="7b3865bd-0448-4c26-b256-4a1c1608a549" containerID="90088b9fecd6d774accc6ab233ac62eb98061694e9f6ccf18b93a2bc68fa4387" exitCode=0 Nov 03 22:12:12.276819 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:12.276368 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" event={"ID":"7b3865bd-0448-4c26-b256-4a1c1608a549","Type":"ContainerDied","Data":"90088b9fecd6d774accc6ab233ac62eb98061694e9f6ccf18b93a2bc68fa4387"} Nov 03 22:12:12.634850 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:12.634757 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:12:12.636245 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:12.636213 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:12:13.283047 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:13.283006 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" event={"ID":"7b3865bd-0448-4c26-b256-4a1c1608a549","Type":"ContainerStarted","Data":"25db7993a41a80df04a90cbc65074327b482298fd490399e87e49c6de1f2b05b"} Nov 03 22:12:13.283443 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:13.283125 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:14.099656 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:14.099591 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" podStartSLOduration=2.910553845 podStartE2EDuration="8.099576147s" totalImagesPullingTime="4.72987324s" totalInitContainerRuntime="459.149062ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:13.519930567 +0000 UTC m=+301.113962535" watchObservedRunningTime="2026-11-03 22:12:14.099576147 +0000 UTC m=+301.693608121" Nov 03 22:12:14.100635 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:14.100617 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cthb4"] Nov 03 22:12:14.104223 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:14.104200 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cthb4" Nov 03 22:12:14.161610 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:14.161573 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:14.161794 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:14.161631 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:14.161794 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:14.161666 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-fvrxq\"" Nov 03 22:12:14.164379 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:14.164349 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cthb4"] Nov 03 22:12:14.204385 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:14.204355 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4dgv\" (UniqueName: \"kubernetes.io/projected/6af404cc-1111-425b-b6a7-ee08f0d1be3b-kube-api-access-z4dgv\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cthb4\" (UID: \"6af404cc-1111-425b-b6a7-ee08f0d1be3b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cthb4" Nov 03 22:12:14.204510 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:14.204407 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6af404cc-1111-425b-b6a7-ee08f0d1be3b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cthb4\" (UID: \"6af404cc-1111-425b-b6a7-ee08f0d1be3b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cthb4" Nov 03 22:12:14.304959 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:14.304919 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6af404cc-1111-425b-b6a7-ee08f0d1be3b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cthb4\" (UID: \"6af404cc-1111-425b-b6a7-ee08f0d1be3b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cthb4" Nov 03 22:12:14.305404 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:14.305011 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4dgv\" (UniqueName: \"kubernetes.io/projected/6af404cc-1111-425b-b6a7-ee08f0d1be3b-kube-api-access-z4dgv\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cthb4\" (UID: \"6af404cc-1111-425b-b6a7-ee08f0d1be3b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cthb4" Nov 03 22:12:14.305404 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:14.305341 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6af404cc-1111-425b-b6a7-ee08f0d1be3b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cthb4\" (UID: \"6af404cc-1111-425b-b6a7-ee08f0d1be3b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cthb4" Nov 03 22:12:14.370676 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:14.370591 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4dgv\" (UniqueName: \"kubernetes.io/projected/6af404cc-1111-425b-b6a7-ee08f0d1be3b-kube-api-access-z4dgv\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cthb4\" (UID: \"6af404cc-1111-425b-b6a7-ee08f0d1be3b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cthb4" Nov 03 22:12:14.416106 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:14.416048 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cthb4" Nov 03 22:12:14.902267 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:14.902233 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cthb4"] Nov 03 22:12:14.905981 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:12:14.905952 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af404cc_1111_425b_b6a7_ee08f0d1be3b.slice/crio-de846e36914127a386733e5bfff91a9f8dca191437c9735de7107f0dd2bb3ff5 WatchSource:0}: Error finding container de846e36914127a386733e5bfff91a9f8dca191437c9735de7107f0dd2bb3ff5: Status 404 returned error can't find the container with id de846e36914127a386733e5bfff91a9f8dca191437c9735de7107f0dd2bb3ff5 Nov 03 22:12:15.295685 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:15.295646 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cthb4" event={"ID":"6af404cc-1111-425b-b6a7-ee08f0d1be3b","Type":"ContainerStarted","Data":"de846e36914127a386733e5bfff91a9f8dca191437c9735de7107f0dd2bb3ff5"} Nov 03 22:12:19.317490 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:19.317452 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cthb4" event={"ID":"6af404cc-1111-425b-b6a7-ee08f0d1be3b","Type":"ContainerStarted","Data":"ac5ff6c11882cae492a94d2c5fb0b7c363414fe5152e79e4113192789191d026"} Nov 03 22:12:19.562691 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:19.562624 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cthb4" podStartSLOduration=1.480639132 podStartE2EDuration="5.562608676s" totalImagesPullingTime="4.081969544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:19.561492071 +0000 UTC m=+307.155524047" watchObservedRunningTime="2026-11-03 22:12:19.562608676 +0000 UTC m=+307.156640650" Nov 03 22:12:21.338677 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:21.338637 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-5jsx7"] Nov 03 22:12:21.342305 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:21.342283 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5jsx7" Nov 03 22:12:21.447079 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:21.447028 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 22:12:21.447266 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:21.447103 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 22:12:21.447266 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:21.447181 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 22:12:21.447266 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:21.447239 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6rlb4\"" Nov 03 22:12:21.448913 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:21.448890 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-5jsx7"] Nov 03 22:12:21.465449 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:21.465411 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp2t7\" (UniqueName: \"kubernetes.io/projected/5cdcb400-fe77-4a4d-84b8-cecb65fe5a61-kube-api-access-pp2t7\") pod \"dns-operator-controller-manager-8557cbdb84-5jsx7\" (UID: \"5cdcb400-fe77-4a4d-84b8-cecb65fe5a61\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5jsx7" Nov 03 22:12:21.565860 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:21.565819 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp2t7\" (UniqueName: \"kubernetes.io/projected/5cdcb400-fe77-4a4d-84b8-cecb65fe5a61-kube-api-access-pp2t7\") pod \"dns-operator-controller-manager-8557cbdb84-5jsx7\" (UID: \"5cdcb400-fe77-4a4d-84b8-cecb65fe5a61\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5jsx7" Nov 03 22:12:21.627792 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:21.627710 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp2t7\" (UniqueName: \"kubernetes.io/projected/5cdcb400-fe77-4a4d-84b8-cecb65fe5a61-kube-api-access-pp2t7\") pod \"dns-operator-controller-manager-8557cbdb84-5jsx7\" (UID: \"5cdcb400-fe77-4a4d-84b8-cecb65fe5a61\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5jsx7" Nov 03 22:12:21.655410 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:21.655363 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5jsx7" Nov 03 22:12:22.038536 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:22.038496 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-5jsx7"] Nov 03 22:12:22.078224 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:12:22.078183 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cdcb400_fe77_4a4d_84b8_cecb65fe5a61.slice/crio-8442cae342e1387c914b1627e4b50bb1e06373d94522cf6b413588a585498f16 WatchSource:0}: Error finding container 8442cae342e1387c914b1627e4b50bb1e06373d94522cf6b413588a585498f16: Status 404 returned error can't find the container with id 8442cae342e1387c914b1627e4b50bb1e06373d94522cf6b413588a585498f16 Nov 03 22:12:22.080724 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:22.080700 5084 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:12:22.332796 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:22.332672 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5jsx7" event={"ID":"5cdcb400-fe77-4a4d-84b8-cecb65fe5a61","Type":"ContainerStarted","Data":"8442cae342e1387c914b1627e4b50bb1e06373d94522cf6b413588a585498f16"} Nov 03 22:12:22.979820 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:22.979782 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-sf9sf"] Nov 03 22:12:22.983177 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:22.983152 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sf9sf" Nov 03 22:12:23.037807 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:23.037763 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-sf9sf"] Nov 03 22:12:23.078366 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:23.078327 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/708fc026-5994-481e-bde0-007c74efd103-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-sf9sf\" (UID: \"708fc026-5994-481e-bde0-007c74efd103\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sf9sf" Nov 03 22:12:23.078535 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:23.078384 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrqqw\" (UniqueName: \"kubernetes.io/projected/708fc026-5994-481e-bde0-007c74efd103-kube-api-access-hrqqw\") pod \"cert-manager-webhook-5dbbbf75cc-sf9sf\" (UID: \"708fc026-5994-481e-bde0-007c74efd103\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sf9sf" Nov 03 22:12:23.090354 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:23.090317 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 22:12:23.090354 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:23.090334 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hdwvk\"" Nov 03 22:12:23.090354 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:23.090336 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 22:12:23.179167 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:23.179123 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/708fc026-5994-481e-bde0-007c74efd103-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-sf9sf\" (UID: \"708fc026-5994-481e-bde0-007c74efd103\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sf9sf" Nov 03 22:12:23.179348 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:23.179182 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrqqw\" (UniqueName: \"kubernetes.io/projected/708fc026-5994-481e-bde0-007c74efd103-kube-api-access-hrqqw\") pod \"cert-manager-webhook-5dbbbf75cc-sf9sf\" (UID: \"708fc026-5994-481e-bde0-007c74efd103\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sf9sf" Nov 03 22:12:23.241046 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:23.240972 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/708fc026-5994-481e-bde0-007c74efd103-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-sf9sf\" (UID: \"708fc026-5994-481e-bde0-007c74efd103\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sf9sf" Nov 03 22:12:23.241583 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:23.241560 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrqqw\" (UniqueName: \"kubernetes.io/projected/708fc026-5994-481e-bde0-007c74efd103-kube-api-access-hrqqw\") pod \"cert-manager-webhook-5dbbbf75cc-sf9sf\" (UID: \"708fc026-5994-481e-bde0-007c74efd103\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sf9sf" Nov 03 22:12:23.308681 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:23.308634 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sf9sf" Nov 03 22:12:23.700808 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:23.700778 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-sf9sf"] Nov 03 22:12:24.292127 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:24.292097 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55b449f4f4-qtdf5" Nov 03 22:12:24.346568 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:24.346526 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sf9sf" event={"ID":"708fc026-5994-481e-bde0-007c74efd103","Type":"ContainerStarted","Data":"b8a3694e2b074c892ccb211093ae5b8679f3653a60aea824eea281864c34edb3"} Nov 03 22:12:26.202931 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:26.202889 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-vjp47"] Nov 03 22:12:26.206505 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:26.206477 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-vjp47" Nov 03 22:12:26.268255 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:26.268228 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-qq4jp\"" Nov 03 22:12:26.309679 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:26.309628 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7mw8\" (UniqueName: \"kubernetes.io/projected/5e81d876-d741-4044-8fdd-dde9acf179a3-kube-api-access-c7mw8\") pod \"authorino-operator-7b7bb4bc77-vjp47\" (UID: \"5e81d876-d741-4044-8fdd-dde9acf179a3\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-vjp47" Nov 03 22:12:26.333576 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:26.333534 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-vjp47"] Nov 03 22:12:26.358681 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:26.358636 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5jsx7" event={"ID":"5cdcb400-fe77-4a4d-84b8-cecb65fe5a61","Type":"ContainerStarted","Data":"3a07324753485c6f8e36db3187dbd5aa048d44bb25c2af069da9a5f1904850ac"} Nov 03 22:12:26.358862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:26.358716 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5jsx7" Nov 03 22:12:26.411172 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:26.411135 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7mw8\" (UniqueName: \"kubernetes.io/projected/5e81d876-d741-4044-8fdd-dde9acf179a3-kube-api-access-c7mw8\") pod \"authorino-operator-7b7bb4bc77-vjp47\" (UID: \"5e81d876-d741-4044-8fdd-dde9acf179a3\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-vjp47" Nov 03 22:12:26.478754 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:26.478709 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7mw8\" (UniqueName: \"kubernetes.io/projected/5e81d876-d741-4044-8fdd-dde9acf179a3-kube-api-access-c7mw8\") pod \"authorino-operator-7b7bb4bc77-vjp47\" (UID: \"5e81d876-d741-4044-8fdd-dde9acf179a3\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-vjp47" Nov 03 22:12:26.498699 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:26.498635 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5jsx7" podStartSLOduration=1.5862467470000001 podStartE2EDuration="5.498619929s" totalImagesPullingTime="3.912373182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:26.497778635 +0000 UTC m=+314.091810608" watchObservedRunningTime="2026-11-03 22:12:26.498619929 +0000 UTC m=+314.092651904" Nov 03 22:12:26.519929 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:26.519843 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-vjp47" Nov 03 22:12:26.909796 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:26.909768 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-vjp47"] Nov 03 22:12:26.912019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:12:26.911989 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e81d876_d741_4044_8fdd_dde9acf179a3.slice/crio-1d661de99276380fbd947a66ad93e04d0bdecf679cd5147251f60d46d274cad0 WatchSource:0}: Error finding container 1d661de99276380fbd947a66ad93e04d0bdecf679cd5147251f60d46d274cad0: Status 404 returned error can't find the container with id 1d661de99276380fbd947a66ad93e04d0bdecf679cd5147251f60d46d274cad0 Nov 03 22:12:27.371243 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:27.371200 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-vjp47" event={"ID":"5e81d876-d741-4044-8fdd-dde9acf179a3","Type":"ContainerStarted","Data":"1d661de99276380fbd947a66ad93e04d0bdecf679cd5147251f60d46d274cad0"} Nov 03 22:12:29.895362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:29.895272 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-lk5r2"] Nov 03 22:12:29.899709 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:29.899684 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lk5r2" Nov 03 22:12:29.957147 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:29.957110 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-lk5r2"] Nov 03 22:12:30.006473 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:30.006436 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:30.006473 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:30.006441 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:30.043785 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:30.043748 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-vzhnh\"" Nov 03 22:12:30.048097 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:30.048055 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxjc9\" (UniqueName: \"kubernetes.io/projected/18e97b9d-a35a-489c-8582-827c031705a4-kube-api-access-kxjc9\") pod \"jobset-operator-674cd59dc5-lk5r2\" (UID: \"18e97b9d-a35a-489c-8582-827c031705a4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lk5r2" Nov 03 22:12:30.048262 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:30.048235 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18e97b9d-a35a-489c-8582-827c031705a4-tmp\") pod \"jobset-operator-674cd59dc5-lk5r2\" (UID: \"18e97b9d-a35a-489c-8582-827c031705a4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lk5r2" Nov 03 22:12:30.149137 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:30.149031 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxjc9\" (UniqueName: \"kubernetes.io/projected/18e97b9d-a35a-489c-8582-827c031705a4-kube-api-access-kxjc9\") pod \"jobset-operator-674cd59dc5-lk5r2\" (UID: \"18e97b9d-a35a-489c-8582-827c031705a4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lk5r2" Nov 03 22:12:30.149137 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:30.149118 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18e97b9d-a35a-489c-8582-827c031705a4-tmp\") pod \"jobset-operator-674cd59dc5-lk5r2\" (UID: \"18e97b9d-a35a-489c-8582-827c031705a4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lk5r2" Nov 03 22:12:30.149485 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:30.149467 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18e97b9d-a35a-489c-8582-827c031705a4-tmp\") pod \"jobset-operator-674cd59dc5-lk5r2\" (UID: \"18e97b9d-a35a-489c-8582-827c031705a4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lk5r2" Nov 03 22:12:30.207274 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:30.207233 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxjc9\" (UniqueName: \"kubernetes.io/projected/18e97b9d-a35a-489c-8582-827c031705a4-kube-api-access-kxjc9\") pod \"jobset-operator-674cd59dc5-lk5r2\" (UID: \"18e97b9d-a35a-489c-8582-827c031705a4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lk5r2" Nov 03 22:12:30.211979 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:30.211946 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lk5r2" Nov 03 22:12:30.388923 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:30.388885 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-vjp47" event={"ID":"5e81d876-d741-4044-8fdd-dde9acf179a3","Type":"ContainerStarted","Data":"2f7d521be2a0194f93ec02d959120b2247ad8129df47ae8f701c3216c8ede2c4"} Nov 03 22:12:30.389127 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:30.389011 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-vjp47" Nov 03 22:12:30.508184 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:30.508056 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-vjp47" podStartSLOduration=1.870175103 podStartE2EDuration="4.508039941s" totalImagesPullingTime="2.637864838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:30.506586397 +0000 UTC m=+318.100618370" watchObservedRunningTime="2026-11-03 22:12:30.508039941 +0000 UTC m=+318.102071915" Nov 03 22:12:30.624958 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:12:30.624923 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18e97b9d_a35a_489c_8582_827c031705a4.slice/crio-caa54175905e5b1c8694cb17ae715f0e93d4716c98b445ac4e321000c24106f0 WatchSource:0}: Error finding container caa54175905e5b1c8694cb17ae715f0e93d4716c98b445ac4e321000c24106f0: Status 404 returned error can't find the container with id caa54175905e5b1c8694cb17ae715f0e93d4716c98b445ac4e321000c24106f0 Nov 03 22:12:30.627501 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:30.627474 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-lk5r2"] Nov 03 22:12:31.395385 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:31.395340 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lk5r2" event={"ID":"18e97b9d-a35a-489c-8582-827c031705a4","Type":"ContainerStarted","Data":"caa54175905e5b1c8694cb17ae715f0e93d4716c98b445ac4e321000c24106f0"} Nov 03 22:12:34.412102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:34.412041 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lk5r2" event={"ID":"18e97b9d-a35a-489c-8582-827c031705a4","Type":"ContainerStarted","Data":"26e71a6550e4d996b4694b7c8bea8de1c1d8f03477a9f60e1376ce7296015bab"} Nov 03 22:12:34.540422 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:34.540360 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lk5r2" podStartSLOduration=2.196841487 podStartE2EDuration="5.540344599s" totalImagesPullingTime="3.343503112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:34.539181406 +0000 UTC m=+322.133213381" watchObservedRunningTime="2026-11-03 22:12:34.540344599 +0000 UTC m=+322.134376569" Nov 03 22:12:34.597238 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:34.597193 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc"] Nov 03 22:12:34.600558 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:34.600527 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc" Nov 03 22:12:34.657167 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:34.657126 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l662p\"" Nov 03 22:12:34.689834 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:34.689736 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89dbe468-1bfc-435e-8c1d-62cf0ccf29ef-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fdgfc\" (UID: \"89dbe468-1bfc-435e-8c1d-62cf0ccf29ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc" Nov 03 22:12:34.689834 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:34.689784 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5zr2\" (UniqueName: \"kubernetes.io/projected/89dbe468-1bfc-435e-8c1d-62cf0ccf29ef-kube-api-access-w5zr2\") pod \"kuadrant-operator-controller-manager-678d87766f-fdgfc\" (UID: \"89dbe468-1bfc-435e-8c1d-62cf0ccf29ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc" Nov 03 22:12:34.706990 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:34.706954 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc"] Nov 03 22:12:34.790318 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:34.790269 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5zr2\" (UniqueName: \"kubernetes.io/projected/89dbe468-1bfc-435e-8c1d-62cf0ccf29ef-kube-api-access-w5zr2\") pod \"kuadrant-operator-controller-manager-678d87766f-fdgfc\" (UID: \"89dbe468-1bfc-435e-8c1d-62cf0ccf29ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc" Nov 03 22:12:34.790527 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:34.790403 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89dbe468-1bfc-435e-8c1d-62cf0ccf29ef-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fdgfc\" (UID: \"89dbe468-1bfc-435e-8c1d-62cf0ccf29ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc" Nov 03 22:12:34.790810 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:34.790788 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89dbe468-1bfc-435e-8c1d-62cf0ccf29ef-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-fdgfc\" (UID: \"89dbe468-1bfc-435e-8c1d-62cf0ccf29ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc" Nov 03 22:12:34.848986 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:34.848940 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5zr2\" (UniqueName: \"kubernetes.io/projected/89dbe468-1bfc-435e-8c1d-62cf0ccf29ef-kube-api-access-w5zr2\") pod \"kuadrant-operator-controller-manager-678d87766f-fdgfc\" (UID: \"89dbe468-1bfc-435e-8c1d-62cf0ccf29ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc" Nov 03 22:12:34.915590 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:34.915552 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc" Nov 03 22:12:35.353488 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:35.353383 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc"] Nov 03 22:12:35.356489 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:12:35.356453 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89dbe468_1bfc_435e_8c1d_62cf0ccf29ef.slice/crio-8f8fddbdc9967f9f39d9136300853ac36043c4d181bf44a152e93e53567ee098 WatchSource:0}: Error finding container 8f8fddbdc9967f9f39d9136300853ac36043c4d181bf44a152e93e53567ee098: Status 404 returned error can't find the container with id 8f8fddbdc9967f9f39d9136300853ac36043c4d181bf44a152e93e53567ee098 Nov 03 22:12:35.418894 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:35.418841 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc" event={"ID":"89dbe468-1bfc-435e-8c1d-62cf0ccf29ef","Type":"ContainerStarted","Data":"8f8fddbdc9967f9f39d9136300853ac36043c4d181bf44a152e93e53567ee098"} Nov 03 22:12:37.373785 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:37.373753 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5jsx7" Nov 03 22:12:39.446871 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:39.446826 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sf9sf" event={"ID":"708fc026-5994-481e-bde0-007c74efd103","Type":"ContainerStarted","Data":"1703747289ba1f384490a002e4028567b60d65d07b20ba3917e41ad879ab224f"} Nov 03 22:12:39.447401 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:39.447022 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sf9sf" Nov 03 22:12:39.565452 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:39.565389 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sf9sf" podStartSLOduration=2.264661811 podStartE2EDuration="17.565370721s" totalImagesPullingTime="15.30070891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:39.564047034 +0000 UTC m=+327.158079009" watchObservedRunningTime="2026-11-03 22:12:39.565370721 +0000 UTC m=+327.159402696" Nov 03 22:12:41.007875 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:41.007836 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5dbbd9b584-gcs6s"] Nov 03 22:12:41.064964 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:41.064922 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-gcs6s" Nov 03 22:12:41.091364 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:41.091323 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5dbbd9b584-gcs6s"] Nov 03 22:12:41.139625 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:41.139589 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 22:12:41.139625 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:41.139584 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 22:12:41.139871 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:41.139668 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-w8jc2\"" Nov 03 22:12:41.253016 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:41.252970 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5daf51f7-c907-41a7-ba73-1846f91f157a-tls-certs\") pod \"kube-auth-proxy-5dbbd9b584-gcs6s\" (UID: \"5daf51f7-c907-41a7-ba73-1846f91f157a\") " pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-gcs6s" Nov 03 22:12:41.253244 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:41.253033 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7mp5\" (UniqueName: \"kubernetes.io/projected/5daf51f7-c907-41a7-ba73-1846f91f157a-kube-api-access-d7mp5\") pod \"kube-auth-proxy-5dbbd9b584-gcs6s\" (UID: \"5daf51f7-c907-41a7-ba73-1846f91f157a\") " pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-gcs6s" Nov 03 22:12:41.253244 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:41.253169 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5daf51f7-c907-41a7-ba73-1846f91f157a-tmp\") pod \"kube-auth-proxy-5dbbd9b584-gcs6s\" (UID: \"5daf51f7-c907-41a7-ba73-1846f91f157a\") " pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-gcs6s" Nov 03 22:12:41.354251 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:41.354212 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5daf51f7-c907-41a7-ba73-1846f91f157a-tls-certs\") pod \"kube-auth-proxy-5dbbd9b584-gcs6s\" (UID: \"5daf51f7-c907-41a7-ba73-1846f91f157a\") " pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-gcs6s" Nov 03 22:12:41.354251 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:41.354263 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7mp5\" (UniqueName: \"kubernetes.io/projected/5daf51f7-c907-41a7-ba73-1846f91f157a-kube-api-access-d7mp5\") pod \"kube-auth-proxy-5dbbd9b584-gcs6s\" (UID: \"5daf51f7-c907-41a7-ba73-1846f91f157a\") " pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-gcs6s" Nov 03 22:12:41.354533 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:41.354311 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5daf51f7-c907-41a7-ba73-1846f91f157a-tmp\") pod \"kube-auth-proxy-5dbbd9b584-gcs6s\" (UID: \"5daf51f7-c907-41a7-ba73-1846f91f157a\") " pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-gcs6s" Nov 03 22:12:41.357094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:41.356962 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5daf51f7-c907-41a7-ba73-1846f91f157a-tmp\") pod \"kube-auth-proxy-5dbbd9b584-gcs6s\" (UID: \"5daf51f7-c907-41a7-ba73-1846f91f157a\") " pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-gcs6s" Nov 03 22:12:41.357395 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:41.357348 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5daf51f7-c907-41a7-ba73-1846f91f157a-tls-certs\") pod \"kube-auth-proxy-5dbbd9b584-gcs6s\" (UID: \"5daf51f7-c907-41a7-ba73-1846f91f157a\") " pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-gcs6s" Nov 03 22:12:41.398768 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:41.398724 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-vjp47" Nov 03 22:12:41.413343 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:41.413310 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7mp5\" (UniqueName: \"kubernetes.io/projected/5daf51f7-c907-41a7-ba73-1846f91f157a-kube-api-access-d7mp5\") pod \"kube-auth-proxy-5dbbd9b584-gcs6s\" (UID: \"5daf51f7-c907-41a7-ba73-1846f91f157a\") " pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-gcs6s" Nov 03 22:12:41.698872 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:41.698838 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-gcs6s" Nov 03 22:12:42.021053 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:12:42.021013 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:12:42.087698 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:42.087665 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5dbbd9b584-gcs6s"] Nov 03 22:12:42.465667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:42.465549 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-gcs6s" event={"ID":"5daf51f7-c907-41a7-ba73-1846f91f157a","Type":"ContainerStarted","Data":"b0b95761cef6b4f1387237459c6f162e44abcc1ad2157a395da2f71b67d3ffc0"} Nov 03 22:12:42.467443 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:42.467410 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc" event={"ID":"89dbe468-1bfc-435e-8c1d-62cf0ccf29ef","Type":"ContainerStarted","Data":"9e7e02666b7dc013420100fe1e13ad297b9703ce7f8509e83195987077edc571"} Nov 03 22:12:42.467608 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:42.467562 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc" Nov 03 22:12:42.593153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:42.593097 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc" podStartSLOduration=2.267881852 podStartE2EDuration="8.593078604s" totalImagesPullingTime="6.325196752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:42.592356171 +0000 UTC m=+330.186388145" watchObservedRunningTime="2026-11-03 22:12:42.593078604 +0000 UTC m=+330.187110571" Nov 03 22:12:45.456154 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:45.456052 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sf9sf" Nov 03 22:12:45.486580 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:45.486544 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-gcs6s" event={"ID":"5daf51f7-c907-41a7-ba73-1846f91f157a","Type":"ContainerStarted","Data":"578796dcd50e60ccf88eb14adadd2a602d8416f929ef47e8269e222f90613930"} Nov 03 22:12:45.692374 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:45.692316 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5dbbd9b584-gcs6s" podStartSLOduration=2.759080731 podStartE2EDuration="5.6922983s" totalImagesPullingTime="2.933217569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:45.6903749 +0000 UTC m=+333.284406873" watchObservedRunningTime="2026-11-03 22:12:45.6922983 +0000 UTC m=+333.286330273" Nov 03 22:12:53.475719 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:12:53.475683 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc" Nov 03 22:13:07.619944 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.619901 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7"] Nov 03 22:13:07.628516 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.628481 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" Nov 03 22:13:07.683518 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.683480 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 22:13:07.683696 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.683527 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6517eb66-69c7-49b0-a3c6-5256dba7034f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-dmvn7\" (UID: \"6517eb66-69c7-49b0-a3c6-5256dba7034f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" Nov 03 22:13:07.683696 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.683566 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6517eb66-69c7-49b0-a3c6-5256dba7034f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-dmvn7\" (UID: \"6517eb66-69c7-49b0-a3c6-5256dba7034f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" Nov 03 22:13:07.683696 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.683646 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6517eb66-69c7-49b0-a3c6-5256dba7034f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-dmvn7\" (UID: \"6517eb66-69c7-49b0-a3c6-5256dba7034f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" Nov 03 22:13:07.683696 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.683665 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pj7m\" (UniqueName: \"kubernetes.io/projected/6517eb66-69c7-49b0-a3c6-5256dba7034f-kube-api-access-4pj7m\") pod \"kserve-module-controller-manager-7566cf5fc8-dmvn7\" (UID: \"6517eb66-69c7-49b0-a3c6-5256dba7034f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" Nov 03 22:13:07.734117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.734048 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-6hgbw\"" Nov 03 22:13:07.734117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.734047 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 22:13:07.784969 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.784930 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6517eb66-69c7-49b0-a3c6-5256dba7034f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-dmvn7\" (UID: \"6517eb66-69c7-49b0-a3c6-5256dba7034f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" Nov 03 22:13:07.785190 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.784987 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6517eb66-69c7-49b0-a3c6-5256dba7034f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-dmvn7\" (UID: \"6517eb66-69c7-49b0-a3c6-5256dba7034f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" Nov 03 22:13:07.785190 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.785040 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6517eb66-69c7-49b0-a3c6-5256dba7034f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-dmvn7\" (UID: \"6517eb66-69c7-49b0-a3c6-5256dba7034f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" Nov 03 22:13:07.785190 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.785098 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pj7m\" (UniqueName: \"kubernetes.io/projected/6517eb66-69c7-49b0-a3c6-5256dba7034f-kube-api-access-4pj7m\") pod \"kserve-module-controller-manager-7566cf5fc8-dmvn7\" (UID: \"6517eb66-69c7-49b0-a3c6-5256dba7034f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" Nov 03 22:13:07.785491 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.785464 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6517eb66-69c7-49b0-a3c6-5256dba7034f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-dmvn7\" (UID: \"6517eb66-69c7-49b0-a3c6-5256dba7034f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" Nov 03 22:13:07.785662 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.785634 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6517eb66-69c7-49b0-a3c6-5256dba7034f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-dmvn7\" (UID: \"6517eb66-69c7-49b0-a3c6-5256dba7034f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" Nov 03 22:13:07.837050 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.836999 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 22:13:07.839848 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.839811 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7"] Nov 03 22:13:07.844763 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.844735 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pj7m\" (UniqueName: \"kubernetes.io/projected/6517eb66-69c7-49b0-a3c6-5256dba7034f-kube-api-access-4pj7m\") pod \"kserve-module-controller-manager-7566cf5fc8-dmvn7\" (UID: \"6517eb66-69c7-49b0-a3c6-5256dba7034f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" Nov 03 22:13:07.846087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.846038 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6517eb66-69c7-49b0-a3c6-5256dba7034f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-dmvn7\" (UID: \"6517eb66-69c7-49b0-a3c6-5256dba7034f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" Nov 03 22:13:07.942651 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:07.942544 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" Nov 03 22:13:08.371243 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:08.371100 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7"] Nov 03 22:13:08.374118 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:13:08.374091 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6517eb66_69c7_49b0_a3c6_5256dba7034f.slice/crio-28d4d324f9c8e8b6c5c79fe89b177a1e0509971853ca13dc9c04cbebfc5c7517 WatchSource:0}: Error finding container 28d4d324f9c8e8b6c5c79fe89b177a1e0509971853ca13dc9c04cbebfc5c7517: Status 404 returned error can't find the container with id 28d4d324f9c8e8b6c5c79fe89b177a1e0509971853ca13dc9c04cbebfc5c7517 Nov 03 22:13:08.603877 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:08.603836 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" event={"ID":"6517eb66-69c7-49b0-a3c6-5256dba7034f","Type":"ContainerStarted","Data":"28d4d324f9c8e8b6c5c79fe89b177a1e0509971853ca13dc9c04cbebfc5c7517"} Nov 03 22:13:10.616222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:10.616180 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" event={"ID":"6517eb66-69c7-49b0-a3c6-5256dba7034f","Type":"ContainerStarted","Data":"d26cf35d24811ed26fe619160ffbec9faeddff10519e1e4f0cb230746a289a5e"} Nov 03 22:13:10.616681 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:10.616283 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" Nov 03 22:13:10.786008 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:10.785951 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" podStartSLOduration=1.74039344 podStartE2EDuration="3.785934679s" totalImagesPullingTime="2.045541239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:10.78473936 +0000 UTC m=+358.378771335" watchObservedRunningTime="2026-11-03 22:13:10.785934679 +0000 UTC m=+358.379966666" Nov 03 22:13:12.022858 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:13:12.022817 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:13:31.622138 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:31.622107 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dmvn7" Nov 03 22:13:33.484699 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:33.484658 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rsbwm"] Nov 03 22:13:33.488388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:33.488364 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rsbwm" Nov 03 22:13:33.542315 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:33.542271 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 22:13:33.542315 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:33.542280 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 22:13:33.593164 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:33.593122 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lh6b2\"" Nov 03 22:13:33.593369 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:33.593305 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 22:13:33.595406 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:33.595379 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rsbwm"] Nov 03 22:13:33.618408 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:33.618362 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-metrics-certs\") pod \"maas-controller-579cf7c6d-rsbwm\" (UID: \"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657\") " pod="opendatahub/maas-controller-579cf7c6d-rsbwm" Nov 03 22:13:33.618408 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:33.618407 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-webhook-cert\") pod \"maas-controller-579cf7c6d-rsbwm\" (UID: \"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657\") " pod="opendatahub/maas-controller-579cf7c6d-rsbwm" Nov 03 22:13:33.618619 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:33.618435 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2phn9\" (UniqueName: \"kubernetes.io/projected/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-kube-api-access-2phn9\") pod \"maas-controller-579cf7c6d-rsbwm\" (UID: \"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657\") " pod="opendatahub/maas-controller-579cf7c6d-rsbwm" Nov 03 22:13:33.719465 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:33.719424 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-metrics-certs\") pod \"maas-controller-579cf7c6d-rsbwm\" (UID: \"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657\") " pod="opendatahub/maas-controller-579cf7c6d-rsbwm" Nov 03 22:13:33.719465 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:33.719467 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-webhook-cert\") pod \"maas-controller-579cf7c6d-rsbwm\" (UID: \"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657\") " pod="opendatahub/maas-controller-579cf7c6d-rsbwm" Nov 03 22:13:33.719683 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:33.719496 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2phn9\" (UniqueName: \"kubernetes.io/projected/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-kube-api-access-2phn9\") pod \"maas-controller-579cf7c6d-rsbwm\" (UID: \"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657\") " pod="opendatahub/maas-controller-579cf7c6d-rsbwm" Nov 03 22:13:33.719683 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:13:33.719619 5084 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Nov 03 22:13:33.719747 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:13:33.719702 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-webhook-cert podName:30d9d9a7-d4b3-4b09-9a26-bf54b97a5657 nodeName:}" failed. No retries permitted until 2026-11-03 22:13:34.219682616 +0000 UTC m=+381.813714575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-webhook-cert") pod "maas-controller-579cf7c6d-rsbwm" (UID: "30d9d9a7-d4b3-4b09-9a26-bf54b97a5657") : secret "maas-controller-webhook-cert" not found Nov 03 22:13:33.722059 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:33.722028 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-metrics-certs\") pod \"maas-controller-579cf7c6d-rsbwm\" (UID: \"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657\") " pod="opendatahub/maas-controller-579cf7c6d-rsbwm" Nov 03 22:13:33.778202 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:33.778165 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2phn9\" (UniqueName: \"kubernetes.io/projected/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-kube-api-access-2phn9\") pod \"maas-controller-579cf7c6d-rsbwm\" (UID: \"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657\") " pod="opendatahub/maas-controller-579cf7c6d-rsbwm" Nov 03 22:13:33.904327 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:33.904287 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57d7bb69fb-scrzg"] Nov 03 22:13:33.907996 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:33.907969 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:13:33.961796 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:33.961763 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57d7bb69fb-scrzg"] Nov 03 22:13:34.022138 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:34.022094 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-webhook-cert\") pod \"maas-controller-57d7bb69fb-scrzg\" (UID: \"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49\") " pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:13:34.022138 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:34.022140 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxbq8\" (UniqueName: \"kubernetes.io/projected/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-kube-api-access-hxbq8\") pod \"maas-controller-57d7bb69fb-scrzg\" (UID: \"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49\") " pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:13:34.022353 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:34.022307 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-metrics-certs\") pod \"maas-controller-57d7bb69fb-scrzg\" (UID: \"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49\") " pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:13:34.122902 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:34.122815 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-metrics-certs\") pod \"maas-controller-57d7bb69fb-scrzg\" (UID: \"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49\") " pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:13:34.123093 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:34.122954 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-webhook-cert\") pod \"maas-controller-57d7bb69fb-scrzg\" (UID: \"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49\") " pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:13:34.123093 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:34.122976 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbq8\" (UniqueName: \"kubernetes.io/projected/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-kube-api-access-hxbq8\") pod \"maas-controller-57d7bb69fb-scrzg\" (UID: \"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49\") " pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:13:34.125453 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:34.125412 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-webhook-cert\") pod \"maas-controller-57d7bb69fb-scrzg\" (UID: \"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49\") " pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:13:34.125623 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:34.125488 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-metrics-certs\") pod \"maas-controller-57d7bb69fb-scrzg\" (UID: \"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49\") " pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:13:34.180919 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:34.180878 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxbq8\" (UniqueName: \"kubernetes.io/projected/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-kube-api-access-hxbq8\") pod \"maas-controller-57d7bb69fb-scrzg\" (UID: \"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49\") " pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:13:34.221774 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:34.221736 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:13:34.223607 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:34.223582 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-webhook-cert\") pod \"maas-controller-579cf7c6d-rsbwm\" (UID: \"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657\") " pod="opendatahub/maas-controller-579cf7c6d-rsbwm" Nov 03 22:13:34.226016 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:34.225983 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-webhook-cert\") pod \"maas-controller-579cf7c6d-rsbwm\" (UID: \"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657\") " pod="opendatahub/maas-controller-579cf7c6d-rsbwm" Nov 03 22:13:34.401728 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:34.401622 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rsbwm" Nov 03 22:13:34.597207 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:34.597180 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57d7bb69fb-scrzg"] Nov 03 22:13:34.599783 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:13:34.599751 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb4c5a2_c6c1_4be9_9656_5eecee18ce49.slice/crio-b4fb5565dbdce58fee2f066dacc8a668a17173f0cb30dc2d01d65a91199a9629 WatchSource:0}: Error finding container b4fb5565dbdce58fee2f066dacc8a668a17173f0cb30dc2d01d65a91199a9629: Status 404 returned error can't find the container with id b4fb5565dbdce58fee2f066dacc8a668a17173f0cb30dc2d01d65a91199a9629 Nov 03 22:13:34.738741 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:34.738647 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" event={"ID":"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49","Type":"ContainerStarted","Data":"b4fb5565dbdce58fee2f066dacc8a668a17173f0cb30dc2d01d65a91199a9629"} Nov 03 22:13:34.777572 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:34.777545 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rsbwm"] Nov 03 22:13:34.779354 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:13:34.779322 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d9d9a7_d4b3_4b09_9a26_bf54b97a5657.slice/crio-4ea4f768d354424f16bc5b85f371f72632ae34e99ec3b02abeb5676cea2f5b2e WatchSource:0}: Error finding container 4ea4f768d354424f16bc5b85f371f72632ae34e99ec3b02abeb5676cea2f5b2e: Status 404 returned error can't find the container with id 4ea4f768d354424f16bc5b85f371f72632ae34e99ec3b02abeb5676cea2f5b2e Nov 03 22:13:35.745404 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:35.745370 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rsbwm" event={"ID":"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657","Type":"ContainerStarted","Data":"4ea4f768d354424f16bc5b85f371f72632ae34e99ec3b02abeb5676cea2f5b2e"} Nov 03 22:13:38.765397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:38.765357 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rsbwm" event={"ID":"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657","Type":"ContainerStarted","Data":"eee66b64d2bfc15579191d5e55a1a4dfb46f98a25d4fd07749899e1e70e94c4e"} Nov 03 22:13:38.765901 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:38.765548 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-rsbwm" Nov 03 22:13:38.767106 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:38.767077 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" event={"ID":"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49","Type":"ContainerStarted","Data":"cc06bd7900e1f2ecc00f506ee29760e024ffdbbb90c42425711b77a816a4819e"} Nov 03 22:13:38.767237 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:38.767218 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:13:38.899406 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:38.899353 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-rsbwm" podStartSLOduration=2.453446103 podStartE2EDuration="5.89933774s" totalImagesPullingTime="3.445891637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:38.897881255 +0000 UTC m=+386.491913229" watchObservedRunningTime="2026-11-03 22:13:38.89933774 +0000 UTC m=+386.493369752" Nov 03 22:13:39.020541 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:39.020433 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" podStartSLOduration=2.306958387 podStartE2EDuration="6.020418267s" totalImagesPullingTime="3.71345988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:39.01895094 +0000 UTC m=+386.612982916" watchObservedRunningTime="2026-11-03 22:13:39.020418267 +0000 UTC m=+386.614450241" Nov 03 22:13:42.023218 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:13:42.023179 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:13:49.774607 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:49.774569 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-rsbwm" Nov 03 22:13:49.775010 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:49.774648 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:13:49.944991 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:49.944953 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rsbwm"] Nov 03 22:13:49.945229 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:49.945205 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-rsbwm" podUID="30d9d9a7-d4b3-4b09-9a26-bf54b97a5657" containerName="manager" containerID="cri-o://eee66b64d2bfc15579191d5e55a1a4dfb46f98a25d4fd07749899e1e70e94c4e" gracePeriod=30 Nov 03 22:13:50.196416 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:50.196385 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rsbwm" Nov 03 22:13:50.263353 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:50.263316 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-metrics-certs\") pod \"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657\" (UID: \"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657\") " Nov 03 22:13:50.263530 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:50.263376 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-webhook-cert\" (UniqueName: \"kubernetes.io/secret/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-webhook-cert\") pod \"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657\" (UID: \"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657\") " Nov 03 22:13:50.265581 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:50.265550 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-webhook-cert" pod "30d9d9a7-d4b3-4b09-9a26-bf54b97a5657" (UID: "30d9d9a7-d4b3-4b09-9a26-bf54b97a5657"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:13:50.265714 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:50.265593 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-metrics-certs" pod "30d9d9a7-d4b3-4b09-9a26-bf54b97a5657" (UID: "30d9d9a7-d4b3-4b09-9a26-bf54b97a5657"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:13:50.364387 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:50.364282 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-kube-api-access-2phn9\" (UniqueName: \"kubernetes.io/projected/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-kube-api-access-2phn9\") pod \"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657\" (UID: \"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657\") " Nov 03 22:13:50.364566 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:50.364493 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:13:50.364566 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:50.364513 5084 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-webhook-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:13:50.366549 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:50.366518 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-kube-api-access-2phn9" pod "30d9d9a7-d4b3-4b09-9a26-bf54b97a5657" (UID: "30d9d9a7-d4b3-4b09-9a26-bf54b97a5657"). InnerVolumeSpecName "kube-api-access-2phn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:13:50.464977 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:50.464927 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2phn9\" (UniqueName: \"kubernetes.io/projected/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657-kube-api-access-2phn9\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:13:50.828954 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:50.828920 5084 generic.go:356] "Generic (PLEG): container finished" podID="30d9d9a7-d4b3-4b09-9a26-bf54b97a5657" containerID="eee66b64d2bfc15579191d5e55a1a4dfb46f98a25d4fd07749899e1e70e94c4e" exitCode=0 Nov 03 22:13:50.829432 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:50.828964 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rsbwm" Nov 03 22:13:50.829432 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:50.829005 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rsbwm" event={"ID":"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657","Type":"ContainerDied","Data":"eee66b64d2bfc15579191d5e55a1a4dfb46f98a25d4fd07749899e1e70e94c4e"} Nov 03 22:13:50.829432 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:50.829091 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rsbwm" event={"ID":"30d9d9a7-d4b3-4b09-9a26-bf54b97a5657","Type":"ContainerDied","Data":"4ea4f768d354424f16bc5b85f371f72632ae34e99ec3b02abeb5676cea2f5b2e"} Nov 03 22:13:50.829432 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:50.829121 5084 scope.go:122] "RemoveContainer" containerID="eee66b64d2bfc15579191d5e55a1a4dfb46f98a25d4fd07749899e1e70e94c4e" Nov 03 22:13:50.838640 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:50.838609 5084 scope.go:122] "RemoveContainer" containerID="eee66b64d2bfc15579191d5e55a1a4dfb46f98a25d4fd07749899e1e70e94c4e" Nov 03 22:13:50.838985 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:13:50.838962 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee66b64d2bfc15579191d5e55a1a4dfb46f98a25d4fd07749899e1e70e94c4e\": container with ID starting with eee66b64d2bfc15579191d5e55a1a4dfb46f98a25d4fd07749899e1e70e94c4e not found: ID does not exist" containerID="eee66b64d2bfc15579191d5e55a1a4dfb46f98a25d4fd07749899e1e70e94c4e" Nov 03 22:13:50.839045 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:50.838997 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee66b64d2bfc15579191d5e55a1a4dfb46f98a25d4fd07749899e1e70e94c4e"} err="failed to get container status \"eee66b64d2bfc15579191d5e55a1a4dfb46f98a25d4fd07749899e1e70e94c4e\": rpc error: code = NotFound desc = could not find container \"eee66b64d2bfc15579191d5e55a1a4dfb46f98a25d4fd07749899e1e70e94c4e\": container with ID starting with eee66b64d2bfc15579191d5e55a1a4dfb46f98a25d4fd07749899e1e70e94c4e not found: ID does not exist" Nov 03 22:13:50.996239 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:50.996151 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rsbwm"] Nov 03 22:13:51.047373 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:51.047328 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rsbwm"] Nov 03 22:13:52.626540 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:13:52.626463 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d9d9a7-d4b3-4b09-9a26-bf54b97a5657" path="/var/lib/kubelet/pods/30d9d9a7-d4b3-4b09-9a26-bf54b97a5657/volumes" Nov 03 22:14:07.231788 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.231747 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz"] Nov 03 22:14:07.232293 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.232171 5084 cpu_manager.go:414] "containerMap: removing container" podUID="30d9d9a7-d4b3-4b09-9a26-bf54b97a5657" containerName="manager" Nov 03 22:14:07.232293 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.232183 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d9d9a7-d4b3-4b09-9a26-bf54b97a5657" containerName="manager" Nov 03 22:14:07.232293 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.232259 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d9d9a7-d4b3-4b09-9a26-bf54b97a5657" containerName="manager" Nov 03 22:14:07.239119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.239092 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.292993 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.292958 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 22:14:07.293304 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.293278 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 22:14:07.293394 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.293364 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 22:14:07.317900 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.317852 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e14a353-5cfa-47ed-805f-ecb9efde013c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.318106 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.317973 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e14a353-5cfa-47ed-805f-ecb9efde013c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.318106 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.318007 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e14a353-5cfa-47ed-805f-ecb9efde013c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.318195 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.318104 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e14a353-5cfa-47ed-805f-ecb9efde013c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.318195 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.318126 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e14a353-5cfa-47ed-805f-ecb9efde013c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.318195 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.318159 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvw94\" (UniqueName: \"kubernetes.io/projected/3e14a353-5cfa-47ed-805f-ecb9efde013c-kube-api-access-gvw94\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.318288 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.318255 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e14a353-5cfa-47ed-805f-ecb9efde013c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.344138 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.344105 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 22:14:07.344349 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.344276 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qcrf7\"" Nov 03 22:14:07.346380 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.346347 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz"] Nov 03 22:14:07.419520 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.419470 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e14a353-5cfa-47ed-805f-ecb9efde013c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.419728 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.419530 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvw94\" (UniqueName: \"kubernetes.io/projected/3e14a353-5cfa-47ed-805f-ecb9efde013c-kube-api-access-gvw94\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.419728 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.419576 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e14a353-5cfa-47ed-805f-ecb9efde013c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.419728 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.419609 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e14a353-5cfa-47ed-805f-ecb9efde013c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.419728 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.419684 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e14a353-5cfa-47ed-805f-ecb9efde013c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.419728 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.419713 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e14a353-5cfa-47ed-805f-ecb9efde013c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.419977 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.419740 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e14a353-5cfa-47ed-805f-ecb9efde013c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.420571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.420546 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e14a353-5cfa-47ed-805f-ecb9efde013c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.422180 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.422153 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e14a353-5cfa-47ed-805f-ecb9efde013c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.422284 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.422223 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e14a353-5cfa-47ed-805f-ecb9efde013c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.422454 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.422433 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e14a353-5cfa-47ed-805f-ecb9efde013c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.422490 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.422465 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e14a353-5cfa-47ed-805f-ecb9efde013c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.479211 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.479168 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e14a353-5cfa-47ed-805f-ecb9efde013c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.479430 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.479407 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvw94\" (UniqueName: \"kubernetes.io/projected/3e14a353-5cfa-47ed-805f-ecb9efde013c-kube-api-access-gvw94\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjlzz\" (UID: \"3e14a353-5cfa-47ed-805f-ecb9efde013c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.551289 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.551248 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:07.986364 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:07.986334 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz"] Nov 03 22:14:07.987907 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:14:07.987881 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e14a353_5cfa_47ed_805f_ecb9efde013c.slice/crio-eeecc61fae0efc380c78105e197be795edcc48c7ccf740e924e295e58a94f027 WatchSource:0}: Error finding container eeecc61fae0efc380c78105e197be795edcc48c7ccf740e924e295e58a94f027: Status 404 returned error can't find the container with id eeecc61fae0efc380c78105e197be795edcc48c7ccf740e924e295e58a94f027 Nov 03 22:14:08.931934 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:08.931889 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" event={"ID":"3e14a353-5cfa-47ed-805f-ecb9efde013c","Type":"ContainerStarted","Data":"eeecc61fae0efc380c78105e197be795edcc48c7ccf740e924e295e58a94f027"} Nov 03 22:14:12.028649 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:14:12.028602 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:14:12.170466 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:12.170431 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:14:12.170589 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:12.170498 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:14:12.597390 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:12.597359 5084 scope.go:122] "RemoveContainer" containerID="18dee5532db44e5169bfd67de9674ff8614ffcadb3f350ea2af54322c700a2a6" Nov 03 22:14:12.960100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:12.959995 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" event={"ID":"3e14a353-5cfa-47ed-805f-ecb9efde013c","Type":"ContainerStarted","Data":"cd84635c8f90fe85e06be565bad27d4dd9897a5acf10fe39f9b7a0b38d0a815c"} Nov 03 22:14:12.960402 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:12.960227 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:13.082974 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:13.082834 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" podStartSLOduration=1.903261449 podStartE2EDuration="6.082814809s" totalImagesPullingTime="4.17955336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:14:13.081405749 +0000 UTC m=+420.675437724" watchObservedRunningTime="2026-11-03 22:14:13.082814809 +0000 UTC m=+420.676846785" Nov 03 22:14:13.967210 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:14:13.967179 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjlzz" Nov 03 22:14:42.021945 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:14:42.021903 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:15:12.023909 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:15:12.023867 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:15:42.021248 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:15:42.021203 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:16:11.583670 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:11.583634 5084 generic.go:356] "Generic (PLEG): container finished" podID="5bb4c5a2-c6c1-4be9-9656-5eecee18ce49" containerID="cc06bd7900e1f2ecc00f506ee29760e024ffdbbb90c42425711b77a816a4819e" exitCode=1 Nov 03 22:16:11.584142 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:11.583706 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" event={"ID":"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49","Type":"ContainerDied","Data":"cc06bd7900e1f2ecc00f506ee29760e024ffdbbb90c42425711b77a816a4819e"} Nov 03 22:16:11.584142 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:11.584103 5084 scope.go:122] "RemoveContainer" containerID="cc06bd7900e1f2ecc00f506ee29760e024ffdbbb90c42425711b77a816a4819e" Nov 03 22:16:12.023300 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:16:12.023262 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:16:13.599570 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:13.599532 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" event={"ID":"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49","Type":"ContainerStarted","Data":"e87355f8b45c32ef5cffc001590054967f8e05bfdadba0cca1f87ca6cb15d7b1"} Nov 03 22:16:13.600054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:13.599760 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:16:24.607654 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:24.607620 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:16:42.022444 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:16:42.022400 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:16:45.637850 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.637813 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl"] Nov 03 22:16:45.641721 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.641698 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.695329 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.695297 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl"] Nov 03 22:16:45.723185 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.723141 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6d1214-9fc7-40a3-930b-1513e14ef268-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zhzjl\" (UID: \"2a6d1214-9fc7-40a3-930b-1513e14ef268\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.723185 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.723185 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72cm9\" (UniqueName: \"kubernetes.io/projected/2a6d1214-9fc7-40a3-930b-1513e14ef268-kube-api-access-72cm9\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zhzjl\" (UID: \"2a6d1214-9fc7-40a3-930b-1513e14ef268\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.723397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.723264 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6d1214-9fc7-40a3-930b-1513e14ef268-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zhzjl\" (UID: \"2a6d1214-9fc7-40a3-930b-1513e14ef268\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.723397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.723338 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6d1214-9fc7-40a3-930b-1513e14ef268-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zhzjl\" (UID: \"2a6d1214-9fc7-40a3-930b-1513e14ef268\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.723471 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.723420 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6d1214-9fc7-40a3-930b-1513e14ef268-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zhzjl\" (UID: \"2a6d1214-9fc7-40a3-930b-1513e14ef268\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.723471 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.723458 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6d1214-9fc7-40a3-930b-1513e14ef268-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zhzjl\" (UID: \"2a6d1214-9fc7-40a3-930b-1513e14ef268\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.744176 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.744141 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2wng2\"" Nov 03 22:16:45.744176 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.744141 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 22:16:45.744176 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.744141 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 22:16:45.744489 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.744215 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 22:16:45.824812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.824765 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6d1214-9fc7-40a3-930b-1513e14ef268-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zhzjl\" (UID: \"2a6d1214-9fc7-40a3-930b-1513e14ef268\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.824812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.824815 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72cm9\" (UniqueName: \"kubernetes.io/projected/2a6d1214-9fc7-40a3-930b-1513e14ef268-kube-api-access-72cm9\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zhzjl\" (UID: \"2a6d1214-9fc7-40a3-930b-1513e14ef268\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.825031 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.824874 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6d1214-9fc7-40a3-930b-1513e14ef268-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zhzjl\" (UID: \"2a6d1214-9fc7-40a3-930b-1513e14ef268\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.825031 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.824902 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6d1214-9fc7-40a3-930b-1513e14ef268-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zhzjl\" (UID: \"2a6d1214-9fc7-40a3-930b-1513e14ef268\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.825031 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.824984 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6d1214-9fc7-40a3-930b-1513e14ef268-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zhzjl\" (UID: \"2a6d1214-9fc7-40a3-930b-1513e14ef268\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.825229 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.825038 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6d1214-9fc7-40a3-930b-1513e14ef268-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zhzjl\" (UID: \"2a6d1214-9fc7-40a3-930b-1513e14ef268\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.825375 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.825347 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6d1214-9fc7-40a3-930b-1513e14ef268-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zhzjl\" (UID: \"2a6d1214-9fc7-40a3-930b-1513e14ef268\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.825446 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.825357 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6d1214-9fc7-40a3-930b-1513e14ef268-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zhzjl\" (UID: \"2a6d1214-9fc7-40a3-930b-1513e14ef268\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.825446 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.825431 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6d1214-9fc7-40a3-930b-1513e14ef268-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zhzjl\" (UID: \"2a6d1214-9fc7-40a3-930b-1513e14ef268\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.827360 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.827339 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6d1214-9fc7-40a3-930b-1513e14ef268-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zhzjl\" (UID: \"2a6d1214-9fc7-40a3-930b-1513e14ef268\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.827655 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.827634 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6d1214-9fc7-40a3-930b-1513e14ef268-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zhzjl\" (UID: \"2a6d1214-9fc7-40a3-930b-1513e14ef268\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.883157 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.883126 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72cm9\" (UniqueName: \"kubernetes.io/projected/2a6d1214-9fc7-40a3-930b-1513e14ef268-kube-api-access-72cm9\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zhzjl\" (UID: \"2a6d1214-9fc7-40a3-930b-1513e14ef268\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:45.955313 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:45.955216 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:46.336413 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:46.336386 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl"] Nov 03 22:16:46.338380 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:16:46.338352 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6d1214_9fc7_40a3_930b_1513e14ef268.slice/crio-24b075c2541f1fd49e8f144ca088dcd962af876763c4ef459cb276166347d160 WatchSource:0}: Error finding container 24b075c2541f1fd49e8f144ca088dcd962af876763c4ef459cb276166347d160: Status 404 returned error can't find the container with id 24b075c2541f1fd49e8f144ca088dcd962af876763c4ef459cb276166347d160 Nov 03 22:16:46.781438 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:46.781396 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" event={"ID":"2a6d1214-9fc7-40a3-930b-1513e14ef268","Type":"ContainerStarted","Data":"24b075c2541f1fd49e8f144ca088dcd962af876763c4ef459cb276166347d160"} Nov 03 22:16:48.794858 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:48.794822 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" event={"ID":"2a6d1214-9fc7-40a3-930b-1513e14ef268","Type":"ContainerStarted","Data":"9769e924f495bbfd280ad46f535542e4c4c851081cd1ba1b6a42d7d92b932cb5"} Nov 03 22:16:48.913120 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:48.913040 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" podStartSLOduration=2.352069556 podStartE2EDuration="3.913023422s" totalImagesPullingTime="1.560953866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:48.911397154 +0000 UTC m=+576.505429128" watchObservedRunningTime="2026-11-03 22:16:48.913023422 +0000 UTC m=+576.507055395" Nov 03 22:16:48.955486 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:48.955442 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:53.104857 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.104769 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qb78n"] Nov 03 22:16:53.108646 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.108623 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.162366 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.162333 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qb78n"] Nov 03 22:16:53.195282 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.195245 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbdfd9d6-132a-423d-b96a-a0119bc8b122-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qb78n\" (UID: \"bbdfd9d6-132a-423d-b96a-a0119bc8b122\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.195433 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.195307 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbdfd9d6-132a-423d-b96a-a0119bc8b122-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qb78n\" (UID: \"bbdfd9d6-132a-423d-b96a-a0119bc8b122\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.195433 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.195365 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbdfd9d6-132a-423d-b96a-a0119bc8b122-home\") pod \"echo-simulator-kserve-64b6f8f96c-qb78n\" (UID: \"bbdfd9d6-132a-423d-b96a-a0119bc8b122\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.195433 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.195394 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbdfd9d6-132a-423d-b96a-a0119bc8b122-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qb78n\" (UID: \"bbdfd9d6-132a-423d-b96a-a0119bc8b122\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.195433 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.195410 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w986w\" (UniqueName: \"kubernetes.io/projected/bbdfd9d6-132a-423d-b96a-a0119bc8b122-kube-api-access-w986w\") pod \"echo-simulator-kserve-64b6f8f96c-qb78n\" (UID: \"bbdfd9d6-132a-423d-b96a-a0119bc8b122\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.195433 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.195431 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbdfd9d6-132a-423d-b96a-a0119bc8b122-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qb78n\" (UID: \"bbdfd9d6-132a-423d-b96a-a0119bc8b122\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.210567 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.210526 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 22:16:53.296434 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.296395 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbdfd9d6-132a-423d-b96a-a0119bc8b122-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qb78n\" (UID: \"bbdfd9d6-132a-423d-b96a-a0119bc8b122\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.296652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.296459 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbdfd9d6-132a-423d-b96a-a0119bc8b122-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qb78n\" (UID: \"bbdfd9d6-132a-423d-b96a-a0119bc8b122\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.296652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.296476 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbdfd9d6-132a-423d-b96a-a0119bc8b122-home\") pod \"echo-simulator-kserve-64b6f8f96c-qb78n\" (UID: \"bbdfd9d6-132a-423d-b96a-a0119bc8b122\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.296652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.296491 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbdfd9d6-132a-423d-b96a-a0119bc8b122-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qb78n\" (UID: \"bbdfd9d6-132a-423d-b96a-a0119bc8b122\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.296652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.296508 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w986w\" (UniqueName: \"kubernetes.io/projected/bbdfd9d6-132a-423d-b96a-a0119bc8b122-kube-api-access-w986w\") pod \"echo-simulator-kserve-64b6f8f96c-qb78n\" (UID: \"bbdfd9d6-132a-423d-b96a-a0119bc8b122\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.296652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.296527 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbdfd9d6-132a-423d-b96a-a0119bc8b122-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qb78n\" (UID: \"bbdfd9d6-132a-423d-b96a-a0119bc8b122\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.296946 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.296859 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbdfd9d6-132a-423d-b96a-a0119bc8b122-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qb78n\" (UID: \"bbdfd9d6-132a-423d-b96a-a0119bc8b122\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.296946 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.296931 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbdfd9d6-132a-423d-b96a-a0119bc8b122-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qb78n\" (UID: \"bbdfd9d6-132a-423d-b96a-a0119bc8b122\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.297051 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.297032 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbdfd9d6-132a-423d-b96a-a0119bc8b122-home\") pod \"echo-simulator-kserve-64b6f8f96c-qb78n\" (UID: \"bbdfd9d6-132a-423d-b96a-a0119bc8b122\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.298862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.298834 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbdfd9d6-132a-423d-b96a-a0119bc8b122-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qb78n\" (UID: \"bbdfd9d6-132a-423d-b96a-a0119bc8b122\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.299096 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.299075 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbdfd9d6-132a-423d-b96a-a0119bc8b122-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qb78n\" (UID: \"bbdfd9d6-132a-423d-b96a-a0119bc8b122\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.359407 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.359330 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w986w\" (UniqueName: \"kubernetes.io/projected/bbdfd9d6-132a-423d-b96a-a0119bc8b122-kube-api-access-w986w\") pod \"echo-simulator-kserve-64b6f8f96c-qb78n\" (UID: \"bbdfd9d6-132a-423d-b96a-a0119bc8b122\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.422373 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.422328 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:16:53.804205 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.804174 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qb78n"] Nov 03 22:16:53.806573 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:16:53.806540 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbdfd9d6_132a_423d_b96a_a0119bc8b122.slice/crio-a095e8a02b7431eea190cd8f4a2fac0896672d51ff167dc5ba44cee2435d03d0 WatchSource:0}: Error finding container a095e8a02b7431eea190cd8f4a2fac0896672d51ff167dc5ba44cee2435d03d0: Status 404 returned error can't find the container with id a095e8a02b7431eea190cd8f4a2fac0896672d51ff167dc5ba44cee2435d03d0 Nov 03 22:16:53.823622 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:53.823581 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" event={"ID":"bbdfd9d6-132a-423d-b96a-a0119bc8b122","Type":"ContainerStarted","Data":"a095e8a02b7431eea190cd8f4a2fac0896672d51ff167dc5ba44cee2435d03d0"} Nov 03 22:16:55.955530 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:55.955407 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:55.972782 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:55.972747 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:56.841863 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:56.841823 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" event={"ID":"bbdfd9d6-132a-423d-b96a-a0119bc8b122","Type":"ContainerStarted","Data":"2d8257240ee6e581e38598b122468516a2c627c9c3cba217b2a07c688eba22f5"} Nov 03 22:16:56.853294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:56.853260 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zhzjl" Nov 03 22:16:56.958564 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:56.958494 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" podStartSLOduration=1.3928540059999999 podStartE2EDuration="3.958474169s" totalImagesPullingTime="2.565620163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:56.957652983 +0000 UTC m=+584.551684955" watchObservedRunningTime="2026-11-03 22:16:56.958474169 +0000 UTC m=+584.552506144" Nov 03 22:16:57.423045 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:16:57.423001 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:17:03.423408 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:03.423363 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:17:03.436388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:03.436353 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:17:03.892717 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:03.892676 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" Nov 03 22:17:09.306690 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.306652 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8"] Nov 03 22:17:09.310565 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.310539 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.335505 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.335458 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.335700 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.335515 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.335700 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.335628 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.335811 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.335711 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.335853 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.335816 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.335853 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.335836 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.335996 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.335978 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.336089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.336023 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.336139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.336105 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk8s5\" (UniqueName: \"kubernetes.io/projected/7d44bd1b-06d9-4452-88e8-78fdae1b2203-kube-api-access-vk8s5\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.364024 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.363985 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kf4rv\"" Nov 03 22:17:09.417560 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.417514 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8"] Nov 03 22:17:09.437090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.437031 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.437299 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.437100 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.437299 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.437251 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.437425 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.437302 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.437425 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.437334 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.437425 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.437350 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.437648 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.437436 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.437648 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.437469 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.437648 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.437503 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk8s5\" (UniqueName: \"kubernetes.io/projected/7d44bd1b-06d9-4452-88e8-78fdae1b2203-kube-api-access-vk8s5\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.437835 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.437799 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.437897 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.437830 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.437982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.437966 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.438032 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.437971 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.438032 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.438026 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.439590 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.439565 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.439889 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.439869 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.495894 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.495858 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.496051 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.496018 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk8s5\" (UniqueName: \"kubernetes.io/projected/7d44bd1b-06d9-4452-88e8-78fdae1b2203-kube-api-access-vk8s5\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-sj8g8\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:09.625295 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:09.625200 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:10.057815 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:10.057788 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8"] Nov 03 22:17:10.058976 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:17:10.058951 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d44bd1b_06d9_4452_88e8_78fdae1b2203.slice/crio-faaee4e0ada9d93436e8df11db0918d2d553b02744bbae66afd271f70a1c5abe WatchSource:0}: Error finding container faaee4e0ada9d93436e8df11db0918d2d553b02744bbae66afd271f70a1c5abe: Status 404 returned error can't find the container with id faaee4e0ada9d93436e8df11db0918d2d553b02744bbae66afd271f70a1c5abe Nov 03 22:17:10.919092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:10.919044 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" event={"ID":"7d44bd1b-06d9-4452-88e8-78fdae1b2203","Type":"ContainerStarted","Data":"faaee4e0ada9d93436e8df11db0918d2d553b02744bbae66afd271f70a1c5abe"} Nov 03 22:17:12.025297 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:17:12.025255 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:17:12.676727 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:12.676696 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:17:12.677437 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:12.677419 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:17:15.296052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:15.296012 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:17:15.296467 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:15.296114 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:17:15.296467 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:15.296144 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:17:15.954154 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:15.954117 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" event={"ID":"7d44bd1b-06d9-4452-88e8-78fdae1b2203","Type":"ContainerStarted","Data":"a1e8ad0d89461866f9c9c9eb1a175d80fdf89fe5f6b714f07d1e0e30ad6e63d1"} Nov 03 22:17:16.176355 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:16.176280 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" podStartSLOduration=1.942281977 podStartE2EDuration="7.176257854s" totalImagesPullingTime="5.233975877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:16.123720616 +0000 UTC m=+603.717752590" watchObservedRunningTime="2026-11-03 22:17:16.176257854 +0000 UTC m=+603.770289831" Nov 03 22:17:16.626422 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:16.626387 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:16.626919 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:16.626896 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:17:16.626971 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:16.626944 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" podUID="7d44bd1b-06d9-4452-88e8-78fdae1b2203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 22:17:17.625663 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:17.625616 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:17:17.625830 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:17.625699 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" podUID="7d44bd1b-06d9-4452-88e8-78fdae1b2203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 22:17:18.625617 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:18.625571 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:17:18.626143 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:18.625644 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" podUID="7d44bd1b-06d9-4452-88e8-78fdae1b2203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 22:17:19.625812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:19.625766 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:17:19.626280 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:19.625850 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" podUID="7d44bd1b-06d9-4452-88e8-78fdae1b2203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 22:17:20.625994 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:20.625953 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:17:20.626420 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:20.626022 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" podUID="7d44bd1b-06d9-4452-88e8-78fdae1b2203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 22:17:21.626193 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:21.626147 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:17:21.626733 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:21.626200 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" podUID="7d44bd1b-06d9-4452-88e8-78fdae1b2203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 22:17:22.625921 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:22.625876 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 03 22:17:22.626152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:22.625931 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" podUID="7d44bd1b-06d9-4452-88e8-78fdae1b2203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 03 22:17:23.769646 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:23.769596 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:23.770154 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:23.770054 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:23.770902 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:23.770879 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:17:24.445015 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.444971 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44"] Nov 03 22:17:24.452438 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.452411 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.574298 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.574259 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44"] Nov 03 22:17:24.581022 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.580988 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.581209 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.581051 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.581209 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.581178 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.581283 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.581249 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.581321 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.581296 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.581354 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.581317 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e541ea68-f573-400b-b0ef-9383d6bcba51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.581396 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.581379 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.581473 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.581460 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.581510 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.581484 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkq4w\" (UniqueName: \"kubernetes.io/projected/e541ea68-f573-400b-b0ef-9383d6bcba51-kube-api-access-pkq4w\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.682706 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.682667 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.682706 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.682715 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.682972 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.682734 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.682972 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.682761 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.682972 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.682805 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e541ea68-f573-400b-b0ef-9383d6bcba51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.682972 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.682832 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.683200 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.682994 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.683200 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.683037 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkq4w\" (UniqueName: \"kubernetes.io/projected/e541ea68-f573-400b-b0ef-9383d6bcba51-kube-api-access-pkq4w\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.683200 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.683143 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.683353 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.683246 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.683353 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.683335 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.683455 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.683386 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.683518 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.683481 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.683655 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.683634 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e541ea68-f573-400b-b0ef-9383d6bcba51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.685468 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.685445 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.685571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.685523 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.740876 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.740796 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.741026 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.740869 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkq4w\" (UniqueName: \"kubernetes.io/projected/e541ea68-f573-400b-b0ef-9383d6bcba51-kube-api-access-pkq4w\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-k2q44\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:24.766805 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:24.766769 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:25.194497 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:25.194463 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44"] Nov 03 22:17:25.196406 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:17:25.196366 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode541ea68_f573_400b_b0ef_9383d6bcba51.slice/crio-83571fb90b798f8fea4c54feee09d9664c3a86108f9e6f3ee031ae45b7309beb WatchSource:0}: Error finding container 83571fb90b798f8fea4c54feee09d9664c3a86108f9e6f3ee031ae45b7309beb: Status 404 returned error can't find the container with id 83571fb90b798f8fea4c54feee09d9664c3a86108f9e6f3ee031ae45b7309beb Nov 03 22:17:25.199511 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:25.199469 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:17:25.199641 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:25.199560 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:17:25.199641 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:25.199611 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:17:26.027259 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:26.027214 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" event={"ID":"e541ea68-f573-400b-b0ef-9383d6bcba51","Type":"ContainerStarted","Data":"fae06f5b57cb11d768172967c4ab7d0d08b3cce62303ec4e5e74a623342606be"} Nov 03 22:17:26.027478 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:26.027276 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" event={"ID":"e541ea68-f573-400b-b0ef-9383d6bcba51","Type":"ContainerStarted","Data":"83571fb90b798f8fea4c54feee09d9664c3a86108f9e6f3ee031ae45b7309beb"} Nov 03 22:17:26.198506 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:26.198445 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" podStartSLOduration=2.198428951 podStartE2EDuration="2.198428951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:26.196290254 +0000 UTC m=+613.790322229" watchObservedRunningTime="2026-11-03 22:17:26.198428951 +0000 UTC m=+613.792460925" Nov 03 22:17:26.780289 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:26.780246 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:26.782001 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:26.781969 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-k2q44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 22:17:26.782096 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:26.782040 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" podUID="e541ea68-f573-400b-b0ef-9383d6bcba51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 22:17:27.778090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:27.776478 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-k2q44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 22:17:27.778090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:27.776567 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" podUID="e541ea68-f573-400b-b0ef-9383d6bcba51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 22:17:27.984289 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:27.984237 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7db955b5ff-gcxs5"] Nov 03 22:17:27.988687 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:27.988648 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7db955b5ff-gcxs5" Nov 03 22:17:28.015999 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:28.015957 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-796rk\" (UniqueName: \"kubernetes.io/projected/8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce-kube-api-access-796rk\") pod \"ai-gateway-controller-7db955b5ff-gcxs5\" (UID: \"8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce\") " pod="opendatahub/ai-gateway-controller-7db955b5ff-gcxs5" Nov 03 22:17:28.017132 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:28.016416 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce-metrics-certs\") pod \"ai-gateway-controller-7db955b5ff-gcxs5\" (UID: \"8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce\") " pod="opendatahub/ai-gateway-controller-7db955b5ff-gcxs5" Nov 03 22:17:28.036623 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:28.036542 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7db955b5ff-gcxs5"] Nov 03 22:17:28.084241 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:28.084205 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 22:17:28.084241 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:28.084207 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-lmm74\"" Nov 03 22:17:28.084241 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:28.084208 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 22:17:28.118572 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:28.118533 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-796rk\" (UniqueName: \"kubernetes.io/projected/8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce-kube-api-access-796rk\") pod \"ai-gateway-controller-7db955b5ff-gcxs5\" (UID: \"8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce\") " pod="opendatahub/ai-gateway-controller-7db955b5ff-gcxs5" Nov 03 22:17:28.118828 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:28.118802 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce-metrics-certs\") pod \"ai-gateway-controller-7db955b5ff-gcxs5\" (UID: \"8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce\") " pod="opendatahub/ai-gateway-controller-7db955b5ff-gcxs5" Nov 03 22:17:28.121356 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:28.121326 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce-metrics-certs\") pod \"ai-gateway-controller-7db955b5ff-gcxs5\" (UID: \"8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce\") " pod="opendatahub/ai-gateway-controller-7db955b5ff-gcxs5" Nov 03 22:17:28.187919 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:28.187884 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-796rk\" (UniqueName: \"kubernetes.io/projected/8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce-kube-api-access-796rk\") pod \"ai-gateway-controller-7db955b5ff-gcxs5\" (UID: \"8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce\") " pod="opendatahub/ai-gateway-controller-7db955b5ff-gcxs5" Nov 03 22:17:28.304087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:28.303955 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7db955b5ff-gcxs5" Nov 03 22:17:28.701213 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:28.701178 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7db955b5ff-gcxs5"] Nov 03 22:17:28.702306 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:17:28.702271 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5b0ee6_dbe7_4062_b447_6f4ccf6ee1ce.slice/crio-0f29e901a27fa6b32c53c7e2c5068c8c60b02cc68524dc7293181c4ded1e9865 WatchSource:0}: Error finding container 0f29e901a27fa6b32c53c7e2c5068c8c60b02cc68524dc7293181c4ded1e9865: Status 404 returned error can't find the container with id 0f29e901a27fa6b32c53c7e2c5068c8c60b02cc68524dc7293181c4ded1e9865 Nov 03 22:17:28.703842 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:28.703823 5084 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:17:28.768998 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:28.768475 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-k2q44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 22:17:28.768998 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:28.768544 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" podUID="e541ea68-f573-400b-b0ef-9383d6bcba51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 22:17:29.050110 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:29.050046 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7db955b5ff-gcxs5" event={"ID":"8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce","Type":"ContainerStarted","Data":"0f29e901a27fa6b32c53c7e2c5068c8c60b02cc68524dc7293181c4ded1e9865"} Nov 03 22:17:29.771557 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:29.771512 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-k2q44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 22:17:29.771876 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:29.771848 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" podUID="e541ea68-f573-400b-b0ef-9383d6bcba51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 22:17:30.780092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:30.777506 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-k2q44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 22:17:30.780092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:30.777577 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" podUID="e541ea68-f573-400b-b0ef-9383d6bcba51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 22:17:31.767673 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:31.767635 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-k2q44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 22:17:31.767863 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:31.767704 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" podUID="e541ea68-f573-400b-b0ef-9383d6bcba51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 22:17:32.767544 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:32.767503 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-k2q44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Nov 03 22:17:32.767937 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:32.767565 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" podUID="e541ea68-f573-400b-b0ef-9383d6bcba51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Nov 03 22:17:33.935039 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:33.934998 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:33.935988 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:33.935902 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:33.937130 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:33.937034 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:17:36.122726 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:36.122682 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7db955b5ff-gcxs5" event={"ID":"8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce","Type":"ContainerStarted","Data":"f4b64e0d1fbdd5216fa9fb90bb67f9c46ec3420a79aedb85e1edb87be7bdd39a"} Nov 03 22:17:36.123185 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:36.122774 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7db955b5ff-gcxs5" Nov 03 22:17:36.238820 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:36.238756 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7db955b5ff-gcxs5" podStartSLOduration=2.664135346 podStartE2EDuration="9.238737619s" totalImagesPullingTime="6.574602273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:36.238184386 +0000 UTC m=+623.832216375" watchObservedRunningTime="2026-11-03 22:17:36.238737619 +0000 UTC m=+623.832769588" Nov 03 22:17:38.434380 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:38.434339 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7q7t4"] Nov 03 22:17:38.438401 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:38.438375 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7q7t4" Nov 03 22:17:38.491929 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:38.491890 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 22:17:38.492183 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:38.491907 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-sdlvs\"" Nov 03 22:17:38.492529 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:38.492498 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7q7t4"] Nov 03 22:17:38.524780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:38.524740 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26z7g\" (UniqueName: \"kubernetes.io/projected/1dedc6fe-4409-4add-b490-1a3b2747520b-kube-api-access-26z7g\") pod \"payload-pre-processing-7b98bb678c-7q7t4\" (UID: \"1dedc6fe-4409-4add-b490-1a3b2747520b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7q7t4" Nov 03 22:17:38.524978 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:38.524809 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dedc6fe-4409-4add-b490-1a3b2747520b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7q7t4\" (UID: \"1dedc6fe-4409-4add-b490-1a3b2747520b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7q7t4" Nov 03 22:17:38.625650 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:38.625602 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26z7g\" (UniqueName: \"kubernetes.io/projected/1dedc6fe-4409-4add-b490-1a3b2747520b-kube-api-access-26z7g\") pod \"payload-pre-processing-7b98bb678c-7q7t4\" (UID: \"1dedc6fe-4409-4add-b490-1a3b2747520b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7q7t4" Nov 03 22:17:38.625833 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:38.625677 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dedc6fe-4409-4add-b490-1a3b2747520b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7q7t4\" (UID: \"1dedc6fe-4409-4add-b490-1a3b2747520b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7q7t4" Nov 03 22:17:38.626437 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:38.626414 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dedc6fe-4409-4add-b490-1a3b2747520b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7q7t4\" (UID: \"1dedc6fe-4409-4add-b490-1a3b2747520b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7q7t4" Nov 03 22:17:38.683914 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:38.683878 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26z7g\" (UniqueName: \"kubernetes.io/projected/1dedc6fe-4409-4add-b490-1a3b2747520b-kube-api-access-26z7g\") pod \"payload-pre-processing-7b98bb678c-7q7t4\" (UID: \"1dedc6fe-4409-4add-b490-1a3b2747520b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7q7t4" Nov 03 22:17:38.752583 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:38.752482 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7q7t4" Nov 03 22:17:39.140822 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:39.140790 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7q7t4"] Nov 03 22:17:39.142956 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:17:39.142919 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dedc6fe_4409_4add_b490_1a3b2747520b.slice/crio-b285a926d9190a7b6c428b0e8ccad39229a705b9c1e631ab3ba072f0137b1757 WatchSource:0}: Error finding container b285a926d9190a7b6c428b0e8ccad39229a705b9c1e631ab3ba072f0137b1757: Status 404 returned error can't find the container with id b285a926d9190a7b6c428b0e8ccad39229a705b9c1e631ab3ba072f0137b1757 Nov 03 22:17:39.160696 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:39.160614 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7q7t4" event={"ID":"1dedc6fe-4409-4add-b490-1a3b2747520b","Type":"ContainerStarted","Data":"b285a926d9190a7b6c428b0e8ccad39229a705b9c1e631ab3ba072f0137b1757"} Nov 03 22:17:42.028904 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:17:42.028863 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:17:43.191180 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:43.191129 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7q7t4" event={"ID":"1dedc6fe-4409-4add-b490-1a3b2747520b","Type":"ContainerStarted","Data":"6c0027ea2b39ae3caf412a8b672f44b8ceeefe54e1f2131a08f92645d4219c13"} Nov 03 22:17:43.191595 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:43.191298 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7q7t4" Nov 03 22:17:43.306742 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:43.306685 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7q7t4" podStartSLOduration=2.006148924 podStartE2EDuration="5.306669806s" totalImagesPullingTime="3.300520882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:43.306142216 +0000 UTC m=+630.900174190" watchObservedRunningTime="2026-11-03 22:17:43.306669806 +0000 UTC m=+630.900701779" Nov 03 22:17:47.131499 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:47.131465 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7db955b5ff-gcxs5" Nov 03 22:17:47.487416 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:47.487326 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6bc9bf8bf6-78nd4"] Nov 03 22:17:47.491437 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:47.491411 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6bc9bf8bf6-78nd4" Nov 03 22:17:47.543911 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:47.543872 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6bc9bf8bf6-78nd4"] Nov 03 22:17:47.613029 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:47.612991 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f959a8d-48e1-4a43-8ef7-ff3bb056a7ff-metrics-certs\") pod \"ai-gateway-controller-6bc9bf8bf6-78nd4\" (UID: \"6f959a8d-48e1-4a43-8ef7-ff3bb056a7ff\") " pod="opendatahub/ai-gateway-controller-6bc9bf8bf6-78nd4" Nov 03 22:17:47.613241 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:47.613097 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vq2r\" (UniqueName: \"kubernetes.io/projected/6f959a8d-48e1-4a43-8ef7-ff3bb056a7ff-kube-api-access-4vq2r\") pod \"ai-gateway-controller-6bc9bf8bf6-78nd4\" (UID: \"6f959a8d-48e1-4a43-8ef7-ff3bb056a7ff\") " pod="opendatahub/ai-gateway-controller-6bc9bf8bf6-78nd4" Nov 03 22:17:47.714474 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:47.714434 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f959a8d-48e1-4a43-8ef7-ff3bb056a7ff-metrics-certs\") pod \"ai-gateway-controller-6bc9bf8bf6-78nd4\" (UID: \"6f959a8d-48e1-4a43-8ef7-ff3bb056a7ff\") " pod="opendatahub/ai-gateway-controller-6bc9bf8bf6-78nd4" Nov 03 22:17:47.714664 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:47.714526 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vq2r\" (UniqueName: \"kubernetes.io/projected/6f959a8d-48e1-4a43-8ef7-ff3bb056a7ff-kube-api-access-4vq2r\") pod \"ai-gateway-controller-6bc9bf8bf6-78nd4\" (UID: \"6f959a8d-48e1-4a43-8ef7-ff3bb056a7ff\") " pod="opendatahub/ai-gateway-controller-6bc9bf8bf6-78nd4" Nov 03 22:17:47.716983 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:47.716955 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f959a8d-48e1-4a43-8ef7-ff3bb056a7ff-metrics-certs\") pod \"ai-gateway-controller-6bc9bf8bf6-78nd4\" (UID: \"6f959a8d-48e1-4a43-8ef7-ff3bb056a7ff\") " pod="opendatahub/ai-gateway-controller-6bc9bf8bf6-78nd4" Nov 03 22:17:47.772422 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:47.772389 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vq2r\" (UniqueName: \"kubernetes.io/projected/6f959a8d-48e1-4a43-8ef7-ff3bb056a7ff-kube-api-access-4vq2r\") pod \"ai-gateway-controller-6bc9bf8bf6-78nd4\" (UID: \"6f959a8d-48e1-4a43-8ef7-ff3bb056a7ff\") " pod="opendatahub/ai-gateway-controller-6bc9bf8bf6-78nd4" Nov 03 22:17:47.804006 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:47.803961 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6bc9bf8bf6-78nd4" Nov 03 22:17:48.180118 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:48.180089 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6bc9bf8bf6-78nd4"] Nov 03 22:17:48.182301 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:17:48.182260 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f959a8d_48e1_4a43_8ef7_ff3bb056a7ff.slice/crio-3be0782da089b7d8c7abf66fccb178d87261b9f6198310608be9f07bbe11b62c WatchSource:0}: Error finding container 3be0782da089b7d8c7abf66fccb178d87261b9f6198310608be9f07bbe11b62c: Status 404 returned error can't find the container with id 3be0782da089b7d8c7abf66fccb178d87261b9f6198310608be9f07bbe11b62c Nov 03 22:17:48.220507 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:48.220470 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6bc9bf8bf6-78nd4" event={"ID":"6f959a8d-48e1-4a43-8ef7-ff3bb056a7ff","Type":"ContainerStarted","Data":"3be0782da089b7d8c7abf66fccb178d87261b9f6198310608be9f07bbe11b62c"} Nov 03 22:17:49.228116 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:49.228078 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6bc9bf8bf6-78nd4" event={"ID":"6f959a8d-48e1-4a43-8ef7-ff3bb056a7ff","Type":"ContainerStarted","Data":"782d1d85660e63fdfb5ab4352ffe11486d78f534f9b84e35145892817336ec58"} Nov 03 22:17:49.228592 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:49.228137 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6bc9bf8bf6-78nd4" Nov 03 22:17:49.345827 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:49.345706 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6bc9bf8bf6-78nd4" podStartSLOduration=1.466756562 podStartE2EDuration="2.345690922s" totalImagesPullingTime="878.93436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:49.343984608 +0000 UTC m=+636.938016580" watchObservedRunningTime="2026-11-03 22:17:49.345690922 +0000 UTC m=+636.939722896" Nov 03 22:17:54.200657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:17:54.200619 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7q7t4" Nov 03 22:18:00.236347 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:00.236312 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6bc9bf8bf6-78nd4" Nov 03 22:18:00.407867 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:00.407823 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7db955b5ff-gcxs5"] Nov 03 22:18:00.408163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:00.408133 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7db955b5ff-gcxs5" podUID="8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce" containerName="manager" containerID="cri-o://f4b64e0d1fbdd5216fa9fb90bb67f9c46ec3420a79aedb85e1edb87be7bdd39a" gracePeriod=30 Nov 03 22:18:00.671626 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:00.671596 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7db955b5ff-gcxs5" Nov 03 22:18:00.837931 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:00.837831 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce-kube-api-access-796rk\" (UniqueName: \"kubernetes.io/projected/8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce-kube-api-access-796rk\") pod \"8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce\" (UID: \"8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce\") " Nov 03 22:18:00.838129 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:00.837997 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce-metrics-certs\") pod \"8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce\" (UID: \"8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce\") " Nov 03 22:18:00.840118 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:00.840084 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce-metrics-certs" pod "8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce" (UID: "8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:18:00.840233 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:00.840213 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce-kube-api-access-796rk" pod "8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce" (UID: "8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce"). InnerVolumeSpecName "kube-api-access-796rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:18:00.939755 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:00.939712 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:18:00.939755 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:00.939748 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-796rk\" (UniqueName: \"kubernetes.io/projected/8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce-kube-api-access-796rk\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:18:01.301468 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:01.301432 5084 generic.go:356] "Generic (PLEG): container finished" podID="8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce" containerID="f4b64e0d1fbdd5216fa9fb90bb67f9c46ec3420a79aedb85e1edb87be7bdd39a" exitCode=0 Nov 03 22:18:01.301997 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:01.301486 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7db955b5ff-gcxs5" Nov 03 22:18:01.301997 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:01.301517 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7db955b5ff-gcxs5" event={"ID":"8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce","Type":"ContainerDied","Data":"f4b64e0d1fbdd5216fa9fb90bb67f9c46ec3420a79aedb85e1edb87be7bdd39a"} Nov 03 22:18:01.301997 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:01.301569 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7db955b5ff-gcxs5" event={"ID":"8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce","Type":"ContainerDied","Data":"0f29e901a27fa6b32c53c7e2c5068c8c60b02cc68524dc7293181c4ded1e9865"} Nov 03 22:18:01.301997 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:01.301593 5084 scope.go:122] "RemoveContainer" containerID="f4b64e0d1fbdd5216fa9fb90bb67f9c46ec3420a79aedb85e1edb87be7bdd39a" Nov 03 22:18:01.311304 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:01.311267 5084 scope.go:122] "RemoveContainer" containerID="f4b64e0d1fbdd5216fa9fb90bb67f9c46ec3420a79aedb85e1edb87be7bdd39a" Nov 03 22:18:01.311647 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:18:01.311614 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b64e0d1fbdd5216fa9fb90bb67f9c46ec3420a79aedb85e1edb87be7bdd39a\": container with ID starting with f4b64e0d1fbdd5216fa9fb90bb67f9c46ec3420a79aedb85e1edb87be7bdd39a not found: ID does not exist" containerID="f4b64e0d1fbdd5216fa9fb90bb67f9c46ec3420a79aedb85e1edb87be7bdd39a" Nov 03 22:18:01.311701 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:01.311648 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b64e0d1fbdd5216fa9fb90bb67f9c46ec3420a79aedb85e1edb87be7bdd39a"} err="failed to get container status \"f4b64e0d1fbdd5216fa9fb90bb67f9c46ec3420a79aedb85e1edb87be7bdd39a\": rpc error: code = NotFound desc = could not find container \"f4b64e0d1fbdd5216fa9fb90bb67f9c46ec3420a79aedb85e1edb87be7bdd39a\": container with ID starting with f4b64e0d1fbdd5216fa9fb90bb67f9c46ec3420a79aedb85e1edb87be7bdd39a not found: ID does not exist" Nov 03 22:18:01.471657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:01.471611 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7db955b5ff-gcxs5"] Nov 03 22:18:01.522499 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:01.522455 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7db955b5ff-gcxs5"] Nov 03 22:18:02.626732 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:02.626692 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce" path="/var/lib/kubelet/pods/8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce/volumes" Nov 03 22:18:12.024573 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:18:12.024525 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:18:30.751883 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.751796 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7"] Nov 03 22:18:30.752401 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.752307 5084 cpu_manager.go:414] "containerMap: removing container" podUID="8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce" containerName="manager" Nov 03 22:18:30.752401 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.752322 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce" containerName="manager" Nov 03 22:18:30.752484 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.752412 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5b0ee6-dbe7-4062-b447-6f4ccf6ee1ce" containerName="manager" Nov 03 22:18:30.755673 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.755653 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.805746 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.805697 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135fb3ee-0298-424f-b188-611bdfbec980-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.806057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.806030 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.806267 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.806249 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135fb3ee-0298-424f-b188-611bdfbec980-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.806442 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.806425 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135fb3ee-0298-424f-b188-611bdfbec980-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.806563 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.806548 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.806701 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.806685 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdg5v\" (UniqueName: \"kubernetes.io/projected/135fb3ee-0298-424f-b188-611bdfbec980-kube-api-access-xdg5v\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.806819 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.806806 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.806962 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.806947 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.807128 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.807082 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.819906 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.819866 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7"] Nov 03 22:18:30.885026 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.884989 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-3378fd-openshift-default-dockercfg-7fv79\"" Nov 03 22:18:30.908281 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.908236 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.908516 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.908363 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135fb3ee-0298-424f-b188-611bdfbec980-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.908516 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.908408 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.908516 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.908436 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135fb3ee-0298-424f-b188-611bdfbec980-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.908516 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.908465 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135fb3ee-0298-424f-b188-611bdfbec980-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.908516 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.908506 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.908775 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.908546 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdg5v\" (UniqueName: \"kubernetes.io/projected/135fb3ee-0298-424f-b188-611bdfbec980-kube-api-access-xdg5v\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.908775 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.908581 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.908775 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.908612 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.909191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.909150 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.909926 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.909892 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.910202 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.910178 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135fb3ee-0298-424f-b188-611bdfbec980-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.910336 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.910315 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.910845 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.910809 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.915877 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.915791 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.916168 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.916144 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135fb3ee-0298-424f-b188-611bdfbec980-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.967718 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.967686 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135fb3ee-0298-424f-b188-611bdfbec980-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:30.968588 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:30.968563 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdg5v\" (UniqueName: \"kubernetes.io/projected/135fb3ee-0298-424f-b188-611bdfbec980-kube-api-access-xdg5v\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:31.069730 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.069686 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:31.510629 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.510539 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7"] Nov 03 22:18:31.552077 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:18:31.552033 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135fb3ee_0298_424f_b188_611bdfbec980.slice/crio-1fa5c9f4ae4d6ca6804b903b14e44f0af2e1eeb5635fe1c5b7066c1c3528bba7 WatchSource:0}: Error finding container 1fa5c9f4ae4d6ca6804b903b14e44f0af2e1eeb5635fe1c5b7066c1c3528bba7: Status 404 returned error can't find the container with id 1fa5c9f4ae4d6ca6804b903b14e44f0af2e1eeb5635fe1c5b7066c1c3528bba7 Nov 03 22:18:31.554220 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.554180 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:18:31.554329 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.554252 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:18:31.554329 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.554279 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:18:31.695130 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.695085 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh"] Nov 03 22:18:31.699539 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.699502 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.717412 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.717377 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2863b4-3169-423e-a0ee-65fd49117624-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.717600 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.717428 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs28v\" (UniqueName: \"kubernetes.io/projected/aa2863b4-3169-423e-a0ee-65fd49117624-kube-api-access-zs28v\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.717600 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.717521 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.717600 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.717551 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.717600 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.717569 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2863b4-3169-423e-a0ee-65fd49117624-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.717815 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.717654 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.717815 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.717708 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2863b4-3169-423e-a0ee-65fd49117624-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.717815 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.717775 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.717964 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.717813 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.754713 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.754648 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f61714-openshift-default-dockercfg-wvcgf\"" Nov 03 22:18:31.781532 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.781488 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh"] Nov 03 22:18:31.821182 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.821149 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2863b4-3169-423e-a0ee-65fd49117624-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.822904 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.822877 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2863b4-3169-423e-a0ee-65fd49117624-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.823000 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.822966 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs28v\" (UniqueName: \"kubernetes.io/projected/aa2863b4-3169-423e-a0ee-65fd49117624-kube-api-access-zs28v\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.823055 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.823018 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.823125 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.823109 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.823180 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.823143 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2863b4-3169-423e-a0ee-65fd49117624-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.823230 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.823203 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.823277 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.823238 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2863b4-3169-423e-a0ee-65fd49117624-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.823329 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.823313 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.823383 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.823353 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.823553 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.823513 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.823768 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.823738 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.823839 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.823814 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.823839 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.823833 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.824020 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.824000 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2863b4-3169-423e-a0ee-65fd49117624-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.826163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.826131 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.891507 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.891468 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs28v\" (UniqueName: \"kubernetes.io/projected/aa2863b4-3169-423e-a0ee-65fd49117624-kube-api-access-zs28v\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:31.892593 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:31.892559 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2863b4-3169-423e-a0ee-65fd49117624-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:32.018056 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.018015 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:32.358043 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.357943 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj"] Nov 03 22:18:32.363076 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.363032 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.416605 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.416561 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-41bb2b-openshift-default-dockercfg-kqcr4\"" Nov 03 22:18:32.416785 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.416758 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj"] Nov 03 22:18:32.430209 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.430167 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892a000a-c16a-4904-8d6d-57952348afc9-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.430365 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.430233 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4znxj\" (UniqueName: \"kubernetes.io/projected/892a000a-c16a-4904-8d6d-57952348afc9-kube-api-access-4znxj\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.430427 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.430365 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.430427 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.430410 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892a000a-c16a-4904-8d6d-57952348afc9-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.430502 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.430436 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.430567 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.430546 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892a000a-c16a-4904-8d6d-57952348afc9-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.430604 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.430582 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.430706 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.430690 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.430765 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.430752 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.483119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.483050 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" event={"ID":"135fb3ee-0298-424f-b188-611bdfbec980","Type":"ContainerStarted","Data":"e7c7e6f5ad3624692c55b68449e4458e1d2cd6548d6d2b5ac253f3cff07756d9"} Nov 03 22:18:32.483295 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.483137 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" event={"ID":"135fb3ee-0298-424f-b188-611bdfbec980","Type":"ContainerStarted","Data":"1fa5c9f4ae4d6ca6804b903b14e44f0af2e1eeb5635fe1c5b7066c1c3528bba7"} Nov 03 22:18:32.533127 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.532429 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.533127 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.532532 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892a000a-c16a-4904-8d6d-57952348afc9-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.533127 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.532579 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4znxj\" (UniqueName: \"kubernetes.io/projected/892a000a-c16a-4904-8d6d-57952348afc9-kube-api-access-4znxj\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.533127 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.532622 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.533127 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.532650 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892a000a-c16a-4904-8d6d-57952348afc9-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.533127 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.532674 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.533127 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.532716 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892a000a-c16a-4904-8d6d-57952348afc9-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.533127 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.532750 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.533127 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.532842 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.533127 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.533016 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh"] Nov 03 22:18:32.533770 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.533316 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.533770 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.533480 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.534150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.534120 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.534701 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.534663 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.534823 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.534750 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892a000a-c16a-4904-8d6d-57952348afc9-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.536381 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.536351 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.537754 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.537729 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892a000a-c16a-4904-8d6d-57952348afc9-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.538536 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:18:32.538513 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2863b4_3169_423e_a0ee_65fd49117624.slice/crio-cd59f94c7ccbf2e638520d36c620c8e63be9e892cecf2918cab40584d17fe33d WatchSource:0}: Error finding container cd59f94c7ccbf2e638520d36c620c8e63be9e892cecf2918cab40584d17fe33d: Status 404 returned error can't find the container with id cd59f94c7ccbf2e638520d36c620c8e63be9e892cecf2918cab40584d17fe33d Nov 03 22:18:32.540933 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.540897 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:18:32.541014 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.540969 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:18:32.541014 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.540998 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:18:32.591101 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.591026 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892a000a-c16a-4904-8d6d-57952348afc9-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.599334 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.599304 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4znxj\" (UniqueName: \"kubernetes.io/projected/892a000a-c16a-4904-8d6d-57952348afc9-kube-api-access-4znxj\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:32.603865 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.603814 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" podStartSLOduration=2.603799211 podStartE2EDuration="2.603799211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:32.601664924 +0000 UTC m=+680.195696890" watchObservedRunningTime="2026-11-03 22:18:32.603799211 +0000 UTC m=+680.197831185" Nov 03 22:18:32.678275 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:32.678170 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:33.027852 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.027807 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r"] Nov 03 22:18:33.033167 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.033140 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.072466 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.072432 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:33.101191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.094881 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r"] Nov 03 22:18:33.101191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.095180 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-6a6b7f-openshift-default-dockercfg-fqnpk\"" Nov 03 22:18:33.101461 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.101431 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:33.141405 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.141363 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.141405 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.141407 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.141631 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.141428 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.141631 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.141588 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.141631 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.141626 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.141806 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.141645 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.141866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.141795 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.141917 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.141873 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.141979 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.141948 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnzbb\" (UniqueName: \"kubernetes.io/projected/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-kube-api-access-rnzbb\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.147000 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.146972 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj"] Nov 03 22:18:33.148555 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:18:33.148510 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892a000a_c16a_4904_8d6d_57952348afc9.slice/crio-042c39debde0478e4cde78e6938ebbf37f9d3b6b1a98548f6a7fddb971459146 WatchSource:0}: Error finding container 042c39debde0478e4cde78e6938ebbf37f9d3b6b1a98548f6a7fddb971459146: Status 404 returned error can't find the container with id 042c39debde0478e4cde78e6938ebbf37f9d3b6b1a98548f6a7fddb971459146 Nov 03 22:18:33.151595 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.151546 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:18:33.151683 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.151641 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:18:33.151723 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.151684 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:18:33.243315 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.243278 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.243507 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.243340 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.243507 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.243394 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.243507 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.243459 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnzbb\" (UniqueName: \"kubernetes.io/projected/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-kube-api-access-rnzbb\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.243695 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.243506 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.243695 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.243538 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.243695 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.243568 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.243695 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.243625 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.243695 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.243660 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.243961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.243741 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.243961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.243789 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.244118 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.244094 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.244176 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.244115 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.244557 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.244536 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.246532 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.246504 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.246661 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.246643 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.303311 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.303227 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.303488 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.303356 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnzbb\" (UniqueName: \"kubernetes.io/projected/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-kube-api-access-rnzbb\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.359054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.359002 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:33.495215 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.495171 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" event={"ID":"892a000a-c16a-4904-8d6d-57952348afc9","Type":"ContainerStarted","Data":"d0bb7ead7a80a2c1536ed2bb792c08d8684a79f21074f865b9febe9ff55c8246"} Nov 03 22:18:33.495427 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.495245 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" event={"ID":"892a000a-c16a-4904-8d6d-57952348afc9","Type":"ContainerStarted","Data":"042c39debde0478e4cde78e6938ebbf37f9d3b6b1a98548f6a7fddb971459146"} Nov 03 22:18:33.499574 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.499531 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" event={"ID":"aa2863b4-3169-423e-a0ee-65fd49117624","Type":"ContainerStarted","Data":"0523d38e5590c7a26a1500ce2c49cd56aabead8e7c904c4a21a66f807c60e87e"} Nov 03 22:18:33.499773 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.499601 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" event={"ID":"aa2863b4-3169-423e-a0ee-65fd49117624","Type":"ContainerStarted","Data":"cd59f94c7ccbf2e638520d36c620c8e63be9e892cecf2918cab40584d17fe33d"} Nov 03 22:18:33.499773 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.499631 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:33.500595 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.500565 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:18:33.616688 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.616580 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" podStartSLOduration=1.616560159 podStartE2EDuration="1.616560159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:33.615735179 +0000 UTC m=+681.209767166" watchObservedRunningTime="2026-11-03 22:18:33.616560159 +0000 UTC m=+681.210592156" Nov 03 22:18:33.679141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.679100 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:33.801278 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.801219 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" podStartSLOduration=2.801197155 podStartE2EDuration="2.801197155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:33.740772865 +0000 UTC m=+681.334804840" watchObservedRunningTime="2026-11-03 22:18:33.801197155 +0000 UTC m=+681.395229130" Nov 03 22:18:33.803516 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.803479 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r"] Nov 03 22:18:33.803891 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:18:33.803850 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2862fdc_74c6_4f14_9a9b_ced8a3d538d5.slice/crio-b2901f46c7231f4b61669c6ac7387c86d26a6e128fbd6f809f12d726606956ed WatchSource:0}: Error finding container b2901f46c7231f4b61669c6ac7387c86d26a6e128fbd6f809f12d726606956ed: Status 404 returned error can't find the container with id b2901f46c7231f4b61669c6ac7387c86d26a6e128fbd6f809f12d726606956ed Nov 03 22:18:33.806845 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.806802 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:18:33.806938 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.806886 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:18:33.806938 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:33.806922 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:18:34.018861 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:34.018817 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:34.024600 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:34.024565 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:34.506390 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:34.506348 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" event={"ID":"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5","Type":"ContainerStarted","Data":"db198584edb5b9186ad6cf094065137653cab298a0d9687f91c065f47de9793e"} Nov 03 22:18:34.506850 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:34.506418 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" event={"ID":"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5","Type":"ContainerStarted","Data":"b2901f46c7231f4b61669c6ac7387c86d26a6e128fbd6f809f12d726606956ed"} Nov 03 22:18:34.510227 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:34.509599 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:34.510906 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:34.510878 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:18:34.627614 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:34.627536 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" podStartSLOduration=2.627515506 podStartE2EDuration="2.627515506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:34.625369243 +0000 UTC m=+682.219401219" watchObservedRunningTime="2026-11-03 22:18:34.627515506 +0000 UTC m=+682.221547678" Nov 03 22:18:34.683792 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:34.683760 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:35.360329 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:35.360287 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:35.366512 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:35.366476 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:35.513788 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:35.513751 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:35.514290 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:35.513811 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:35.514517 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:35.514494 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:18:35.514609 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:35.514593 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:18:42.023622 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:18:42.023565 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:18:58.403657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:58.403618 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr"] Nov 03 22:18:58.407861 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:58.407829 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr" Nov 03 22:18:58.463374 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:58.463332 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr"] Nov 03 22:18:58.508381 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:58.508346 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f61714\"" Nov 03 22:18:58.508620 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:58.508456 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f61714-dockercfg-2k27n\"" Nov 03 22:18:58.550282 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:58.550238 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbbe6f8-7037-43d3-b3e3-0e48b6316632-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr\" (UID: \"9fbbe6f8-7037-43d3-b3e3-0e48b6316632\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr" Nov 03 22:18:58.550492 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:58.550378 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm2nx\" (UniqueName: \"kubernetes.io/projected/9fbbe6f8-7037-43d3-b3e3-0e48b6316632-kube-api-access-jm2nx\") pod \"payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr\" (UID: \"9fbbe6f8-7037-43d3-b3e3-0e48b6316632\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr" Nov 03 22:18:58.651576 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:58.651523 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbbe6f8-7037-43d3-b3e3-0e48b6316632-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr\" (UID: \"9fbbe6f8-7037-43d3-b3e3-0e48b6316632\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr" Nov 03 22:18:58.651786 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:58.651607 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2nx\" (UniqueName: \"kubernetes.io/projected/9fbbe6f8-7037-43d3-b3e3-0e48b6316632-kube-api-access-jm2nx\") pod \"payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr\" (UID: \"9fbbe6f8-7037-43d3-b3e3-0e48b6316632\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr" Nov 03 22:18:58.652341 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:58.652314 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbbe6f8-7037-43d3-b3e3-0e48b6316632-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr\" (UID: \"9fbbe6f8-7037-43d3-b3e3-0e48b6316632\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr" Nov 03 22:18:58.710475 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:58.710389 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2nx\" (UniqueName: \"kubernetes.io/projected/9fbbe6f8-7037-43d3-b3e3-0e48b6316632-kube-api-access-jm2nx\") pod \"payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr\" (UID: \"9fbbe6f8-7037-43d3-b3e3-0e48b6316632\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr" Nov 03 22:18:58.728346 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:58.728306 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr" Nov 03 22:18:59.115097 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:59.115033 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr"] Nov 03 22:18:59.116728 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:18:59.116697 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbbe6f8_7037_43d3_b3e3_0e48b6316632.slice/crio-5ae596a15ea1c6f8ab9a49fc95428fd4bb7353402e674e347ec2faf85d8f699e WatchSource:0}: Error finding container 5ae596a15ea1c6f8ab9a49fc95428fd4bb7353402e674e347ec2faf85d8f699e: Status 404 returned error can't find the container with id 5ae596a15ea1c6f8ab9a49fc95428fd4bb7353402e674e347ec2faf85d8f699e Nov 03 22:18:59.717763 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:59.717721 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr" event={"ID":"9fbbe6f8-7037-43d3-b3e3-0e48b6316632","Type":"ContainerStarted","Data":"75f3fa604b2e6c35d734eb58327c129665c28e2ed936a1e9ffabcc98da99bd17"} Nov 03 22:18:59.718279 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:59.717778 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr" event={"ID":"9fbbe6f8-7037-43d3-b3e3-0e48b6316632","Type":"ContainerStarted","Data":"5ae596a15ea1c6f8ab9a49fc95428fd4bb7353402e674e347ec2faf85d8f699e"} Nov 03 22:18:59.718279 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:18:59.717810 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr" Nov 03 22:19:00.725955 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:00.725843 5084 generic.go:356] "Generic (PLEG): container finished" podID="135fb3ee-0298-424f-b188-611bdfbec980" containerID="e7c7e6f5ad3624692c55b68449e4458e1d2cd6548d6d2b5ac253f3cff07756d9" exitCode=0 Nov 03 22:19:00.725955 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:00.725919 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" event={"ID":"135fb3ee-0298-424f-b188-611bdfbec980","Type":"ContainerDied","Data":"e7c7e6f5ad3624692c55b68449e4458e1d2cd6548d6d2b5ac253f3cff07756d9"} Nov 03 22:19:00.726464 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:00.726383 5084 scope.go:122] "RemoveContainer" containerID="e7c7e6f5ad3624692c55b68449e4458e1d2cd6548d6d2b5ac253f3cff07756d9" Nov 03 22:19:00.727641 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:00.727609 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:19:00.727751 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:00.727679 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:19:00.727751 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:00.727711 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:19:00.850436 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:00.850379 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr" podStartSLOduration=2.8503647340000002 podStartE2EDuration="2.850364734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:59.837387543 +0000 UTC m=+707.431419516" watchObservedRunningTime="2026-11-03 22:19:00.850364734 +0000 UTC m=+708.444396707" Nov 03 22:19:01.733688 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:01.733641 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" event={"ID":"135fb3ee-0298-424f-b188-611bdfbec980","Type":"ContainerStarted","Data":"c2dcd4eedb3bcd36b3f68503ec93e2582e62530a7bed4928e11440a3317282d4"} Nov 03 22:19:02.073509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:02.073467 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:19:02.077260 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:02.077214 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:19:02.077578 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:02.077548 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" podUID="135fb3ee-0298-424f-b188-611bdfbec980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 22:19:03.133703 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:03.133650 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:19:03.134260 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:03.133840 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" podUID="135fb3ee-0298-424f-b188-611bdfbec980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 22:19:03.508762 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:03.506203 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx"] Nov 03 22:19:03.512052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:03.512017 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:19:03.512228 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:03.512146 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" Nov 03 22:19:03.570099 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:03.548725 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kclt9\" (UniqueName: \"kubernetes.io/projected/1a5c9a91-01a9-40e6-80f0-daa26a323c21-kube-api-access-kclt9\") pod \"payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx\" (UID: \"1a5c9a91-01a9-40e6-80f0-daa26a323c21\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" Nov 03 22:19:03.570099 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:03.548798 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a5c9a91-01a9-40e6-80f0-daa26a323c21-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx\" (UID: \"1a5c9a91-01a9-40e6-80f0-daa26a323c21\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" Nov 03 22:19:03.570099 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:03.562895 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx"] Nov 03 22:19:03.603088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:03.602517 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-f7e6db-dockercfg-htx7h\"" Nov 03 22:19:03.603088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:03.602604 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-f7e6db\"" Nov 03 22:19:03.651749 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:03.649823 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kclt9\" (UniqueName: \"kubernetes.io/projected/1a5c9a91-01a9-40e6-80f0-daa26a323c21-kube-api-access-kclt9\") pod \"payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx\" (UID: \"1a5c9a91-01a9-40e6-80f0-daa26a323c21\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" Nov 03 22:19:03.651749 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:03.649932 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a5c9a91-01a9-40e6-80f0-daa26a323c21-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx\" (UID: \"1a5c9a91-01a9-40e6-80f0-daa26a323c21\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" Nov 03 22:19:03.652055 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:03.652021 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a5c9a91-01a9-40e6-80f0-daa26a323c21-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx\" (UID: \"1a5c9a91-01a9-40e6-80f0-daa26a323c21\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" Nov 03 22:19:03.717277 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:03.717243 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kclt9\" (UniqueName: \"kubernetes.io/projected/1a5c9a91-01a9-40e6-80f0-daa26a323c21-kube-api-access-kclt9\") pod \"payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx\" (UID: \"1a5c9a91-01a9-40e6-80f0-daa26a323c21\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" Nov 03 22:19:03.829758 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:03.829159 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" Nov 03 22:19:04.074102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:04.071512 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:19:04.074102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:04.071576 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" podUID="135fb3ee-0298-424f-b188-611bdfbec980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 22:19:04.300091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:04.294271 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx"] Nov 03 22:19:04.753771 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:04.753584 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" event={"ID":"1a5c9a91-01a9-40e6-80f0-daa26a323c21","Type":"ContainerStarted","Data":"a8c83a5d73d3d730c6c6fdb86e513b1a1a42c3b2061d5000817cfab2e681dccd"} Nov 03 22:19:04.753771 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:04.753640 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" event={"ID":"1a5c9a91-01a9-40e6-80f0-daa26a323c21","Type":"ContainerStarted","Data":"9e7ff5fff18f8ac631bd1d03c454336f0755f752fb985c34756cc7dd01affebc"} Nov 03 22:19:04.753771 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:04.753689 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" Nov 03 22:19:04.874600 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:04.874524 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" podStartSLOduration=1.8745042939999998 podStartE2EDuration="1.874504294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:19:04.87236777 +0000 UTC m=+712.466399745" watchObservedRunningTime="2026-11-03 22:19:04.874504294 +0000 UTC m=+712.468536275" Nov 03 22:19:05.070390 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:05.070351 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:19:05.070736 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:05.070681 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" podUID="135fb3ee-0298-424f-b188-611bdfbec980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 22:19:06.073091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:06.070541 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:19:06.073091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:06.070611 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" podUID="135fb3ee-0298-424f-b188-611bdfbec980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 22:19:07.072929 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:07.072478 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 22:19:07.072929 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:07.072546 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" podUID="135fb3ee-0298-424f-b188-611bdfbec980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 22:19:08.075342 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:08.075304 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:19:08.076761 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:08.076736 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:19:10.729782 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:10.729687 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr" Nov 03 22:19:10.820676 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:10.820587 5084 generic.go:356] "Generic (PLEG): container finished" podID="892a000a-c16a-4904-8d6d-57952348afc9" containerID="d0bb7ead7a80a2c1536ed2bb792c08d8684a79f21074f865b9febe9ff55c8246" exitCode=0 Nov 03 22:19:10.820676 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:10.820660 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" event={"ID":"892a000a-c16a-4904-8d6d-57952348afc9","Type":"ContainerDied","Data":"d0bb7ead7a80a2c1536ed2bb792c08d8684a79f21074f865b9febe9ff55c8246"} Nov 03 22:19:10.821144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:10.821125 5084 scope.go:122] "RemoveContainer" containerID="d0bb7ead7a80a2c1536ed2bb792c08d8684a79f21074f865b9febe9ff55c8246" Nov 03 22:19:10.822575 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:10.822535 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:19:10.822654 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:10.822619 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:19:10.822691 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:10.822659 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:19:11.830850 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:11.830806 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" event={"ID":"892a000a-c16a-4904-8d6d-57952348afc9","Type":"ContainerStarted","Data":"1383950ff17acdec04cfb6f1f9e4cffb3c04762f4e5f64f657b253764fc9e172"} Nov 03 22:19:12.024437 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:19:12.024402 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:19:12.680089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:12.679788 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:19:12.680743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:12.680698 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 22:19:12.680878 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:12.680763 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" podUID="892a000a-c16a-4904-8d6d-57952348afc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 22:19:13.680534 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:13.680491 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 22:19:13.680964 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:13.680555 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" podUID="892a000a-c16a-4904-8d6d-57952348afc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 22:19:14.680574 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:14.680526 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 22:19:14.681164 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:14.680634 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" podUID="892a000a-c16a-4904-8d6d-57952348afc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 22:19:15.037354 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.037304 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4"] Nov 03 22:19:15.041698 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.041655 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:19:15.097690 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.097638 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4"] Nov 03 22:19:15.134561 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.134524 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4d71d8-dfc0-437a-a639-8e763b69f249-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4\" (UID: \"4e4d71d8-dfc0-437a-a639-8e763b69f249\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:19:15.134561 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.134574 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4d71d8-dfc0-437a-a639-8e763b69f249-metrics-certs\") pod \"maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4\" (UID: \"4e4d71d8-dfc0-437a-a639-8e763b69f249\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:19:15.134795 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.134677 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4d71d8-dfc0-437a-a639-8e763b69f249-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4\" (UID: \"4e4d71d8-dfc0-437a-a639-8e763b69f249\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:19:15.134795 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.134769 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrngq\" (UniqueName: \"kubernetes.io/projected/4e4d71d8-dfc0-437a-a639-8e763b69f249-kube-api-access-mrngq\") pod \"maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4\" (UID: \"4e4d71d8-dfc0-437a-a639-8e763b69f249\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:19:15.144964 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.144915 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-f61714-serving-cert\"" Nov 03 22:19:15.144964 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.144958 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:19:15.145293 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.145006 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 22:19:15.145293 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.145181 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:19:15.146089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.146048 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vjv49\"" Nov 03 22:19:15.236324 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.236285 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4d71d8-dfc0-437a-a639-8e763b69f249-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4\" (UID: \"4e4d71d8-dfc0-437a-a639-8e763b69f249\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:19:15.236514 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.236373 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrngq\" (UniqueName: \"kubernetes.io/projected/4e4d71d8-dfc0-437a-a639-8e763b69f249-kube-api-access-mrngq\") pod \"maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4\" (UID: \"4e4d71d8-dfc0-437a-a639-8e763b69f249\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:19:15.236514 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.236425 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4d71d8-dfc0-437a-a639-8e763b69f249-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4\" (UID: \"4e4d71d8-dfc0-437a-a639-8e763b69f249\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:19:15.236514 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.236499 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4d71d8-dfc0-437a-a639-8e763b69f249-metrics-certs\") pod \"maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4\" (UID: \"4e4d71d8-dfc0-437a-a639-8e763b69f249\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:19:15.236623 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:19:15.236537 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-serving-cert: secret "maas-api-e2e-worker-w0-f61714-serving-cert" not found Nov 03 22:19:15.236623 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:19:15.236613 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4d71d8-dfc0-437a-a639-8e763b69f249-maas-api-tls podName:4e4d71d8-dfc0-437a-a639-8e763b69f249 nodeName:}" failed. No retries permitted until 2026-11-03 22:19:15.736592395 +0000 UTC m=+723.330624370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e4d71d8-dfc0-437a-a639-8e763b69f249-maas-api-tls") pod "maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" (UID: "4e4d71d8-dfc0-437a-a639-8e763b69f249") : secret "maas-api-e2e-worker-w0-f61714-serving-cert" not found Nov 03 22:19:15.236982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.236962 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4d71d8-dfc0-437a-a639-8e763b69f249-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4\" (UID: \"4e4d71d8-dfc0-437a-a639-8e763b69f249\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:19:15.239232 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.239206 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4d71d8-dfc0-437a-a639-8e763b69f249-metrics-certs\") pod \"maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4\" (UID: \"4e4d71d8-dfc0-437a-a639-8e763b69f249\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:19:15.305183 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.301355 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrngq\" (UniqueName: \"kubernetes.io/projected/4e4d71d8-dfc0-437a-a639-8e763b69f249-kube-api-access-mrngq\") pod \"maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4\" (UID: \"4e4d71d8-dfc0-437a-a639-8e763b69f249\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:19:15.680151 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.679319 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 22:19:15.680151 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.679382 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" podUID="892a000a-c16a-4904-8d6d-57952348afc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 22:19:15.744816 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.744777 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4d71d8-dfc0-437a-a639-8e763b69f249-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4\" (UID: \"4e4d71d8-dfc0-437a-a639-8e763b69f249\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:19:15.747890 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.747854 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4d71d8-dfc0-437a-a639-8e763b69f249-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4\" (UID: \"4e4d71d8-dfc0-437a-a639-8e763b69f249\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:19:15.787516 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.787479 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" Nov 03 22:19:15.960701 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:15.960605 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:19:16.400732 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:16.389473 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4"] Nov 03 22:19:16.680592 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:16.680481 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 22:19:16.680592 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:16.680554 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" podUID="892a000a-c16a-4904-8d6d-57952348afc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 22:19:16.879706 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:16.879666 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" event={"ID":"4e4d71d8-dfc0-437a-a639-8e763b69f249","Type":"ContainerStarted","Data":"3ebccfe542d977c9fc81512de387babde9ee6a64858c3256d48c506f0e4cec92"} Nov 03 22:19:17.684315 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:17.684273 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:19:17.684590 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:17.684568 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:19:17.685326 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:17.685306 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:19:18.903902 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:18.903855 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" event={"ID":"4e4d71d8-dfc0-437a-a639-8e763b69f249","Type":"ContainerStarted","Data":"3028118b05fb3004bd2e0b373180a9c5805fe763b863052b4bc695774559eef9"} Nov 03 22:19:18.904443 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:18.904360 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:19:19.041491 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:19.041365 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" podStartSLOduration=2.689854983 podStartE2EDuration="5.041351119s" totalImagesPullingTime="2.351496136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:19:19.040868105 +0000 UTC m=+726.634900079" watchObservedRunningTime="2026-11-03 22:19:19.041351119 +0000 UTC m=+726.635383094" Nov 03 22:19:24.917176 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:24.917141 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:19:31.029042 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:31.029005 5084 generic.go:356] "Generic (PLEG): container finished" podID="f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" containerID="db198584edb5b9186ad6cf094065137653cab298a0d9687f91c065f47de9793e" exitCode=0 Nov 03 22:19:31.029578 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:31.029096 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" event={"ID":"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5","Type":"ContainerDied","Data":"db198584edb5b9186ad6cf094065137653cab298a0d9687f91c065f47de9793e"} Nov 03 22:19:31.029578 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:31.029496 5084 scope.go:122] "RemoveContainer" containerID="db198584edb5b9186ad6cf094065137653cab298a0d9687f91c065f47de9793e" Nov 03 22:19:31.030964 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:31.030933 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:19:31.031031 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:31.031004 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:19:31.031098 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:31.031037 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:19:32.040380 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:32.040337 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" event={"ID":"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5","Type":"ContainerStarted","Data":"e6fa413bffd0164d8e9dda994e28d080418a7b3b598c4a051b8424c7449ad819"} Nov 03 22:19:32.364148 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:32.362436 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:19:32.366291 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:32.366255 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 22:19:32.366566 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:32.366538 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" podUID="f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 22:19:33.368091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:33.367489 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 22:19:33.368091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:33.367560 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" podUID="f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 22:19:34.363570 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:34.363527 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 22:19:34.363891 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:34.363861 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" podUID="f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 22:19:35.373316 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:35.373259 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 22:19:35.374007 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:35.373937 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" podUID="f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 22:19:35.516298 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:35.516248 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:19:36.369091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:36.367208 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 22:19:36.369091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:36.367279 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" podUID="f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 22:19:37.367092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:37.363494 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 22:19:37.367092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:37.363562 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" podUID="f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 22:19:38.365267 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:38.365223 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:19:38.366497 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:38.366468 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:19:41.143925 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:41.143885 5084 generic.go:356] "Generic (PLEG): container finished" podID="aa2863b4-3169-423e-a0ee-65fd49117624" containerID="0523d38e5590c7a26a1500ce2c49cd56aabead8e7c904c4a21a66f807c60e87e" exitCode=0 Nov 03 22:19:41.144625 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:41.143965 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" event={"ID":"aa2863b4-3169-423e-a0ee-65fd49117624","Type":"ContainerDied","Data":"0523d38e5590c7a26a1500ce2c49cd56aabead8e7c904c4a21a66f807c60e87e"} Nov 03 22:19:41.144625 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:41.144419 5084 scope.go:122] "RemoveContainer" containerID="0523d38e5590c7a26a1500ce2c49cd56aabead8e7c904c4a21a66f807c60e87e" Nov 03 22:19:41.145875 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:41.145847 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:19:41.145937 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:41.145916 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:19:41.145975 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:41.145962 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:19:42.027414 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:19:42.027366 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:19:42.152336 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:42.152300 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" event={"ID":"aa2863b4-3169-423e-a0ee-65fd49117624","Type":"ContainerStarted","Data":"60f124f6b353f0ae2aa2400e506f6441d968f2dfea4df98c3f0e425bae44889b"} Nov 03 22:19:43.018794 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:43.018746 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:19:43.019106 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:43.019049 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:19:43.019222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:43.019156 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" podUID="aa2863b4-3169-423e-a0ee-65fd49117624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 22:19:44.019351 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:44.019314 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:19:44.019751 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:44.019372 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" podUID="aa2863b4-3169-423e-a0ee-65fd49117624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 22:19:45.018833 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:45.018797 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:19:45.019144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:45.018852 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" podUID="aa2863b4-3169-423e-a0ee-65fd49117624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 22:19:46.019405 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:46.019359 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:19:46.019869 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:46.019417 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" podUID="aa2863b4-3169-423e-a0ee-65fd49117624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 22:19:47.019186 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:47.019143 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 22:19:47.019409 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:47.019209 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" podUID="aa2863b4-3169-423e-a0ee-65fd49117624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 22:19:48.023562 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:48.023521 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:19:48.024083 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:48.024025 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:19:48.024928 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:19:48.024905 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:20:12.024540 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:20:12.024498 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:20:42.021892 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:20:42.021855 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:21:12.046567 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:21:12.046522 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:21:42.023207 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:21:42.023168 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:22:12.026783 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:22:12.026741 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:22:12.731177 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:12.731145 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:22:12.732429 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:12.732400 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:22:18.025786 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.025748 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc"] Nov 03 22:22:18.029974 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.029943 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.083563 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.083518 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-f81f8f-openshift-default-dockercfg-mvg86\"" Nov 03 22:22:18.139250 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.139206 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc"] Nov 03 22:22:18.146184 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.146141 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.146357 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.146298 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.146510 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.146481 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.146587 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.146524 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.146639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.146583 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n5sb\" (UniqueName: \"kubernetes.io/projected/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-kube-api-access-4n5sb\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.146639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.146609 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.146743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.146652 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.146743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.146704 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.146743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.146727 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.247918 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.247874 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.248199 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.247931 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.248199 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.247952 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.248199 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.247996 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.248199 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.248101 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.248199 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.248144 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.248199 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.248161 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.248532 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.248210 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n5sb\" (UniqueName: \"kubernetes.io/projected/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-kube-api-access-4n5sb\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.248532 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.248236 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.248532 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.248482 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.248629 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.248562 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.248629 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.248594 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.248695 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.248651 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.248798 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.248775 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.250615 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.250582 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.250941 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.250914 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.306308 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.306209 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.306308 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.306259 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n5sb\" (UniqueName: \"kubernetes.io/projected/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-kube-api-access-4n5sb\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.345048 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.345009 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:18.782688 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.782658 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc"] Nov 03 22:22:18.784245 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:22:18.784219 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231a9925_fa0d_4a64_b4e6_cdeaf59d44be.slice/crio-aad23be0317ec6b7436f6f301edaeab505da363583c4f85b0620de458baa5fa3 WatchSource:0}: Error finding container aad23be0317ec6b7436f6f301edaeab505da363583c4f85b0620de458baa5fa3: Status 404 returned error can't find the container with id aad23be0317ec6b7436f6f301edaeab505da363583c4f85b0620de458baa5fa3 Nov 03 22:22:18.786556 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.786522 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:22:18.786628 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.786592 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:22:18.786668 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:18.786642 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:22:19.178570 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:19.178471 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" event={"ID":"231a9925-fa0d-4a64-b4e6-cdeaf59d44be","Type":"ContainerStarted","Data":"58dae898ea1d44e88837d3579719a44179245956841753c3b907bedb73cfc496"} Nov 03 22:22:19.178570 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:19.178550 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" event={"ID":"231a9925-fa0d-4a64-b4e6-cdeaf59d44be","Type":"ContainerStarted","Data":"aad23be0317ec6b7436f6f301edaeab505da363583c4f85b0620de458baa5fa3"} Nov 03 22:22:19.298277 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:19.298210 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" podStartSLOduration=2.298189709 podStartE2EDuration="2.298189709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:19.296666213 +0000 UTC m=+906.890698189" watchObservedRunningTime="2026-11-03 22:22:19.298189709 +0000 UTC m=+906.892221685" Nov 03 22:22:19.345431 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:19.345390 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:19.347124 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:19.347093 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:22:19.347266 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:19.347146 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" podUID="231a9925-fa0d-4a64-b4e6-cdeaf59d44be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 22:22:20.346332 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:20.346293 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:22:20.346757 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:20.346364 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" podUID="231a9925-fa0d-4a64-b4e6-cdeaf59d44be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 22:22:21.345467 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:21.345428 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:22:21.345666 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:21.345487 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" podUID="231a9925-fa0d-4a64-b4e6-cdeaf59d44be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 22:22:22.345585 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:22.345546 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:22:22.346012 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:22.345605 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" podUID="231a9925-fa0d-4a64-b4e6-cdeaf59d44be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 22:22:23.345957 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:23.345917 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:22:23.346434 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:23.345977 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" podUID="231a9925-fa0d-4a64-b4e6-cdeaf59d44be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 22:22:24.346045 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:24.345995 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:22:24.346499 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:24.346106 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" podUID="231a9925-fa0d-4a64-b4e6-cdeaf59d44be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 22:22:25.345449 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:25.345411 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 22:22:25.345631 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:25.345484 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" podUID="231a9925-fa0d-4a64-b4e6-cdeaf59d44be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 22:22:26.351025 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:26.350993 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:26.351542 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:26.351307 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:26.352139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:22:26.352115 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:22:42.023916 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:22:42.023878 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:23:03.350380 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:03.350337 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2"] Nov 03 22:23:03.354348 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:03.354317 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2" Nov 03 22:23:03.405894 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:03.405855 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2"] Nov 03 22:23:03.426634 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:03.426585 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88770b16-bcfd2\" (UID: \"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2" Nov 03 22:23:03.426849 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:03.426782 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88770b16-bcfd2\" (UID: \"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2" Nov 03 22:23:03.426929 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:03.426891 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzjlf\" (UniqueName: \"kubernetes.io/projected/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-kube-api-access-fzjlf\") pod \"maas-api-revoke-sub-88770b16-bcfd2\" (UID: \"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2" Nov 03 22:23:03.454234 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:03.454188 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:23:03.454448 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:03.454265 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:23:03.527950 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:03.527900 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88770b16-bcfd2\" (UID: \"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2" Nov 03 22:23:03.528156 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:03.528015 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88770b16-bcfd2\" (UID: \"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2" Nov 03 22:23:03.528156 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:03.528119 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzjlf\" (UniqueName: \"kubernetes.io/projected/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-kube-api-access-fzjlf\") pod \"maas-api-revoke-sub-88770b16-bcfd2\" (UID: \"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2" Nov 03 22:23:03.528761 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:03.528740 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88770b16-bcfd2\" (UID: \"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2" Nov 03 22:23:03.530496 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:03.530475 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88770b16-bcfd2\" (UID: \"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2" Nov 03 22:23:03.586041 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:03.586000 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzjlf\" (UniqueName: \"kubernetes.io/projected/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-kube-api-access-fzjlf\") pod \"maas-api-revoke-sub-88770b16-bcfd2\" (UID: \"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2" Nov 03 22:23:03.669967 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:03.669869 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2" Nov 03 22:23:04.053701 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:04.053669 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2"] Nov 03 22:23:04.055750 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:23:04.055715 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c98c7dd_fdfc_4dc7_a7cc_c81e30cd376d.slice/crio-422753bfe6be9008acc113450fa61fd91f078ded583ebd49d5ae9554b696be9d WatchSource:0}: Error finding container 422753bfe6be9008acc113450fa61fd91f078ded583ebd49d5ae9554b696be9d: Status 404 returned error can't find the container with id 422753bfe6be9008acc113450fa61fd91f078ded583ebd49d5ae9554b696be9d Nov 03 22:23:04.057733 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:04.057710 5084 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:23:04.519142 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:04.519101 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2" event={"ID":"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d","Type":"ContainerStarted","Data":"422753bfe6be9008acc113450fa61fd91f078ded583ebd49d5ae9554b696be9d"} Nov 03 22:23:10.563663 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:10.563618 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2" event={"ID":"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d","Type":"ContainerStarted","Data":"9e2c47f0f893e2d64c1f3427868bebd26932027da0cb3395a577b4031217b11d"} Nov 03 22:23:10.680345 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:10.680292 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2" podStartSLOduration=1.8565695679999998 podStartE2EDuration="7.680277484s" totalImagesPullingTime="5.823707916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:10.679393954 +0000 UTC m=+958.273425929" watchObservedRunningTime="2026-11-03 22:23:10.680277484 +0000 UTC m=+958.274309457" Nov 03 22:23:11.573143 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:11.573106 5084 generic.go:356] "Generic (PLEG): container finished" podID="2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d" containerID="9e2c47f0f893e2d64c1f3427868bebd26932027da0cb3395a577b4031217b11d" exitCode=0 Nov 03 22:23:11.573543 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:11.573246 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2" event={"ID":"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d","Type":"ContainerDied","Data":"9e2c47f0f893e2d64c1f3427868bebd26932027da0cb3395a577b4031217b11d"} Nov 03 22:23:12.025358 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:23:12.025317 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:23:12.750434 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:12.750403 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2" Nov 03 22:23:12.829288 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:12.829246 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-kube-api-access-fzjlf\" (UniqueName: \"kubernetes.io/projected/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-kube-api-access-fzjlf\") pod \"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d\" (UID: \"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d\") " Nov 03 22:23:12.829464 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:12.829305 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-maas-api-openshift-ca\") pod \"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d\" (UID: \"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d\") " Nov 03 22:23:12.829464 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:12.829333 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-maas-api-xks-ca\") pod \"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d\" (UID: \"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d\") " Nov 03 22:23:12.829769 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:12.829732 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-maas-api-openshift-ca" pod "2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d" (UID: "2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:12.831720 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:12.831652 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-maas-api-xks-ca" pod "2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d" (UID: "2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:12.831720 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:12.831698 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-kube-api-access-fzjlf" pod "2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d" (UID: "2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d"). InnerVolumeSpecName "kube-api-access-fzjlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:12.931186 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:12.931140 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzjlf\" (UniqueName: \"kubernetes.io/projected/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-kube-api-access-fzjlf\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:12.931186 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:12.931173 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-maas-api-openshift-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:12.931186 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:12.931184 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d-maas-api-xks-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:13.587627 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:13.587597 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2" Nov 03 22:23:13.587846 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:13.587624 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2" event={"ID":"2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d","Type":"ContainerDied","Data":"422753bfe6be9008acc113450fa61fd91f078ded583ebd49d5ae9554b696be9d"} Nov 03 22:23:13.587846 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:13.587682 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="422753bfe6be9008acc113450fa61fd91f078ded583ebd49d5ae9554b696be9d" Nov 03 22:23:30.923285 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:30.923243 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8"] Nov 03 22:23:30.923830 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:30.923753 5084 cpu_manager.go:414] "containerMap: removing container" podUID="2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d" containerName="revoke-keys" Nov 03 22:23:30.923830 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:30.923768 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d" containerName="revoke-keys" Nov 03 22:23:30.923915 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:30.923840 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d" containerName="revoke-keys" Nov 03 22:23:30.930207 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:30.930184 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8" Nov 03 22:23:30.959142 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:30.959103 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15ee554-111c-4eff-bfa0-98c40634e126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c74e75a1-j6bk8\" (UID: \"a15ee554-111c-4eff-bfa0-98c40634e126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8" Nov 03 22:23:30.959353 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:30.959159 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2stk\" (UniqueName: \"kubernetes.io/projected/a15ee554-111c-4eff-bfa0-98c40634e126-kube-api-access-h2stk\") pod \"maas-api-revoke-sub-c74e75a1-j6bk8\" (UID: \"a15ee554-111c-4eff-bfa0-98c40634e126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8" Nov 03 22:23:30.959353 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:30.959272 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15ee554-111c-4eff-bfa0-98c40634e126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c74e75a1-j6bk8\" (UID: \"a15ee554-111c-4eff-bfa0-98c40634e126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8" Nov 03 22:23:30.981916 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:30.981871 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8"] Nov 03 22:23:31.029003 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:31.028969 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:23:31.029224 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:31.029093 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:23:31.060977 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:31.060923 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15ee554-111c-4eff-bfa0-98c40634e126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c74e75a1-j6bk8\" (UID: \"a15ee554-111c-4eff-bfa0-98c40634e126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8" Nov 03 22:23:31.061209 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:31.061007 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2stk\" (UniqueName: \"kubernetes.io/projected/a15ee554-111c-4eff-bfa0-98c40634e126-kube-api-access-h2stk\") pod \"maas-api-revoke-sub-c74e75a1-j6bk8\" (UID: \"a15ee554-111c-4eff-bfa0-98c40634e126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8" Nov 03 22:23:31.061209 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:31.061086 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15ee554-111c-4eff-bfa0-98c40634e126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c74e75a1-j6bk8\" (UID: \"a15ee554-111c-4eff-bfa0-98c40634e126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8" Nov 03 22:23:31.062024 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:31.061994 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15ee554-111c-4eff-bfa0-98c40634e126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c74e75a1-j6bk8\" (UID: \"a15ee554-111c-4eff-bfa0-98c40634e126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8" Nov 03 22:23:31.063616 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:31.063592 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15ee554-111c-4eff-bfa0-98c40634e126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c74e75a1-j6bk8\" (UID: \"a15ee554-111c-4eff-bfa0-98c40634e126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8" Nov 03 22:23:31.120576 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:31.120534 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2stk\" (UniqueName: \"kubernetes.io/projected/a15ee554-111c-4eff-bfa0-98c40634e126-kube-api-access-h2stk\") pod \"maas-api-revoke-sub-c74e75a1-j6bk8\" (UID: \"a15ee554-111c-4eff-bfa0-98c40634e126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8" Nov 03 22:23:31.244115 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:31.243994 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8" Nov 03 22:23:31.628237 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:31.628203 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8"] Nov 03 22:23:31.629250 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:23:31.629219 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15ee554_111c_4eff_bfa0_98c40634e126.slice/crio-a376653b2d3b48f1b58543e0d291611abdc3e44d5d74356ca5962c968345f244 WatchSource:0}: Error finding container a376653b2d3b48f1b58543e0d291611abdc3e44d5d74356ca5962c968345f244: Status 404 returned error can't find the container with id a376653b2d3b48f1b58543e0d291611abdc3e44d5d74356ca5962c968345f244 Nov 03 22:23:31.715251 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:31.715212 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8" event={"ID":"a15ee554-111c-4eff-bfa0-98c40634e126","Type":"ContainerStarted","Data":"a376653b2d3b48f1b58543e0d291611abdc3e44d5d74356ca5962c968345f244"} Nov 03 22:23:32.722965 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:32.722932 5084 generic.go:356] "Generic (PLEG): container finished" podID="a15ee554-111c-4eff-bfa0-98c40634e126" containerID="c4eb8918afdc5c2a4e2c2991edd6edca0835f2916d76823630f934a54e233388" exitCode=0 Nov 03 22:23:32.723439 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:32.723018 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8" event={"ID":"a15ee554-111c-4eff-bfa0-98c40634e126","Type":"ContainerDied","Data":"c4eb8918afdc5c2a4e2c2991edd6edca0835f2916d76823630f934a54e233388"} Nov 03 22:23:33.885369 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:33.885336 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8" Nov 03 22:23:33.991212 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:33.991102 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a15ee554-111c-4eff-bfa0-98c40634e126-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15ee554-111c-4eff-bfa0-98c40634e126-maas-api-xks-ca\") pod \"a15ee554-111c-4eff-bfa0-98c40634e126\" (UID: \"a15ee554-111c-4eff-bfa0-98c40634e126\") " Nov 03 22:23:33.991388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:33.991245 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a15ee554-111c-4eff-bfa0-98c40634e126-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15ee554-111c-4eff-bfa0-98c40634e126-maas-api-openshift-ca\") pod \"a15ee554-111c-4eff-bfa0-98c40634e126\" (UID: \"a15ee554-111c-4eff-bfa0-98c40634e126\") " Nov 03 22:23:33.991388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:33.991271 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15ee554-111c-4eff-bfa0-98c40634e126-kube-api-access-h2stk\" (UniqueName: \"kubernetes.io/projected/a15ee554-111c-4eff-bfa0-98c40634e126-kube-api-access-h2stk\") pod \"a15ee554-111c-4eff-bfa0-98c40634e126\" (UID: \"a15ee554-111c-4eff-bfa0-98c40634e126\") " Nov 03 22:23:33.991674 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:33.991650 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15ee554-111c-4eff-bfa0-98c40634e126-maas-api-openshift-ca" pod "a15ee554-111c-4eff-bfa0-98c40634e126" (UID: "a15ee554-111c-4eff-bfa0-98c40634e126"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:33.993386 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:33.993336 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a15ee554-111c-4eff-bfa0-98c40634e126-maas-api-xks-ca" pod "a15ee554-111c-4eff-bfa0-98c40634e126" (UID: "a15ee554-111c-4eff-bfa0-98c40634e126"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:33.993500 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:33.993459 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15ee554-111c-4eff-bfa0-98c40634e126-kube-api-access-h2stk" pod "a15ee554-111c-4eff-bfa0-98c40634e126" (UID: "a15ee554-111c-4eff-bfa0-98c40634e126"). InnerVolumeSpecName "kube-api-access-h2stk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:34.092536 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:34.092497 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15ee554-111c-4eff-bfa0-98c40634e126-maas-api-xks-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:34.092536 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:34.092533 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15ee554-111c-4eff-bfa0-98c40634e126-maas-api-openshift-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:34.092536 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:34.092544 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2stk\" (UniqueName: \"kubernetes.io/projected/a15ee554-111c-4eff-bfa0-98c40634e126-kube-api-access-h2stk\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:34.737131 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:34.737097 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8" Nov 03 22:23:34.737316 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:34.737097 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8" event={"ID":"a15ee554-111c-4eff-bfa0-98c40634e126","Type":"ContainerDied","Data":"a376653b2d3b48f1b58543e0d291611abdc3e44d5d74356ca5962c968345f244"} Nov 03 22:23:34.737316 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:34.737233 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a376653b2d3b48f1b58543e0d291611abdc3e44d5d74356ca5962c968345f244" Nov 03 22:23:42.024233 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:23:42.024190 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:23:46.796743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:46.796699 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v"] Nov 03 22:23:46.797217 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:46.797202 5084 cpu_manager.go:414] "containerMap: removing container" podUID="a15ee554-111c-4eff-bfa0-98c40634e126" containerName="revoke-keys" Nov 03 22:23:46.797264 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:46.797218 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15ee554-111c-4eff-bfa0-98c40634e126" containerName="revoke-keys" Nov 03 22:23:46.797324 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:46.797314 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15ee554-111c-4eff-bfa0-98c40634e126" containerName="revoke-keys" Nov 03 22:23:46.800948 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:46.800926 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v" Nov 03 22:23:46.849823 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:46.849778 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfb1096-ae10-4197-b046-ae9a310706e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-806f55be-m949v\" (UID: \"fbfb1096-ae10-4197-b046-ae9a310706e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v" Nov 03 22:23:46.850142 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:46.850111 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-294df\" (UniqueName: \"kubernetes.io/projected/fbfb1096-ae10-4197-b046-ae9a310706e8-kube-api-access-294df\") pod \"maas-api-revoke-sub-806f55be-m949v\" (UID: \"fbfb1096-ae10-4197-b046-ae9a310706e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v" Nov 03 22:23:46.850290 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:46.850236 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfb1096-ae10-4197-b046-ae9a310706e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-806f55be-m949v\" (UID: \"fbfb1096-ae10-4197-b046-ae9a310706e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v" Nov 03 22:23:46.855049 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:46.855013 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v"] Nov 03 22:23:46.901054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:46.901011 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:23:46.901298 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:46.901269 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:23:46.951014 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:46.950973 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-294df\" (UniqueName: \"kubernetes.io/projected/fbfb1096-ae10-4197-b046-ae9a310706e8-kube-api-access-294df\") pod \"maas-api-revoke-sub-806f55be-m949v\" (UID: \"fbfb1096-ae10-4197-b046-ae9a310706e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v" Nov 03 22:23:46.951263 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:46.951055 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfb1096-ae10-4197-b046-ae9a310706e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-806f55be-m949v\" (UID: \"fbfb1096-ae10-4197-b046-ae9a310706e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v" Nov 03 22:23:46.951263 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:46.951143 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfb1096-ae10-4197-b046-ae9a310706e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-806f55be-m949v\" (UID: \"fbfb1096-ae10-4197-b046-ae9a310706e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v" Nov 03 22:23:46.951816 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:46.951791 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfb1096-ae10-4197-b046-ae9a310706e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-806f55be-m949v\" (UID: \"fbfb1096-ae10-4197-b046-ae9a310706e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v" Nov 03 22:23:46.953615 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:46.953591 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfb1096-ae10-4197-b046-ae9a310706e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-806f55be-m949v\" (UID: \"fbfb1096-ae10-4197-b046-ae9a310706e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v" Nov 03 22:23:47.010276 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:47.010243 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-294df\" (UniqueName: \"kubernetes.io/projected/fbfb1096-ae10-4197-b046-ae9a310706e8-kube-api-access-294df\") pod \"maas-api-revoke-sub-806f55be-m949v\" (UID: \"fbfb1096-ae10-4197-b046-ae9a310706e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v" Nov 03 22:23:47.115340 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:47.115248 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v" Nov 03 22:23:47.502777 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:47.502743 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v"] Nov 03 22:23:47.504860 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:23:47.504817 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbfb1096_ae10_4197_b046_ae9a310706e8.slice/crio-cdde1fdd5b00b9bd8a8594795bfd5dcf92c945c547e39a856f30d207f84ecd10 WatchSource:0}: Error finding container cdde1fdd5b00b9bd8a8594795bfd5dcf92c945c547e39a856f30d207f84ecd10: Status 404 returned error can't find the container with id cdde1fdd5b00b9bd8a8594795bfd5dcf92c945c547e39a856f30d207f84ecd10 Nov 03 22:23:47.827166 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:47.827123 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v" event={"ID":"fbfb1096-ae10-4197-b046-ae9a310706e8","Type":"ContainerStarted","Data":"3891cc2c5c18701d1ab78f7997fb0466a346de77520c6f556e3dad23757aea5a"} Nov 03 22:23:47.827621 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:47.827179 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v" event={"ID":"fbfb1096-ae10-4197-b046-ae9a310706e8","Type":"ContainerStarted","Data":"cdde1fdd5b00b9bd8a8594795bfd5dcf92c945c547e39a856f30d207f84ecd10"} Nov 03 22:23:47.945002 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:47.944940 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v" podStartSLOduration=1.944924568 podStartE2EDuration="1.944924568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:47.942212584 +0000 UTC m=+995.536244586" watchObservedRunningTime="2026-11-03 22:23:47.944924568 +0000 UTC m=+995.538956542" Nov 03 22:23:48.835524 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:48.835494 5084 generic.go:356] "Generic (PLEG): container finished" podID="fbfb1096-ae10-4197-b046-ae9a310706e8" containerID="3891cc2c5c18701d1ab78f7997fb0466a346de77520c6f556e3dad23757aea5a" exitCode=0 Nov 03 22:23:48.835962 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:48.835572 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v" event={"ID":"fbfb1096-ae10-4197-b046-ae9a310706e8","Type":"ContainerDied","Data":"3891cc2c5c18701d1ab78f7997fb0466a346de77520c6f556e3dad23757aea5a"} Nov 03 22:23:50.007829 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:50.007795 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v" Nov 03 22:23:50.085038 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:50.084992 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbfb1096-ae10-4197-b046-ae9a310706e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfb1096-ae10-4197-b046-ae9a310706e8-maas-api-openshift-ca\") pod \"fbfb1096-ae10-4197-b046-ae9a310706e8\" (UID: \"fbfb1096-ae10-4197-b046-ae9a310706e8\") " Nov 03 22:23:50.085260 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:50.085058 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbfb1096-ae10-4197-b046-ae9a310706e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfb1096-ae10-4197-b046-ae9a310706e8-maas-api-xks-ca\") pod \"fbfb1096-ae10-4197-b046-ae9a310706e8\" (UID: \"fbfb1096-ae10-4197-b046-ae9a310706e8\") " Nov 03 22:23:50.085260 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:50.085234 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbfb1096-ae10-4197-b046-ae9a310706e8-kube-api-access-294df\" (UniqueName: \"kubernetes.io/projected/fbfb1096-ae10-4197-b046-ae9a310706e8-kube-api-access-294df\") pod \"fbfb1096-ae10-4197-b046-ae9a310706e8\" (UID: \"fbfb1096-ae10-4197-b046-ae9a310706e8\") " Nov 03 22:23:50.085538 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:50.085507 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbfb1096-ae10-4197-b046-ae9a310706e8-maas-api-openshift-ca" pod "fbfb1096-ae10-4197-b046-ae9a310706e8" (UID: "fbfb1096-ae10-4197-b046-ae9a310706e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:23:50.085699 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:50.085670 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfb1096-ae10-4197-b046-ae9a310706e8-maas-api-openshift-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:50.087418 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:50.087383 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfb1096-ae10-4197-b046-ae9a310706e8-kube-api-access-294df" pod "fbfb1096-ae10-4197-b046-ae9a310706e8" (UID: "fbfb1096-ae10-4197-b046-ae9a310706e8"). InnerVolumeSpecName "kube-api-access-294df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:23:50.087621 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:50.087588 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbfb1096-ae10-4197-b046-ae9a310706e8-maas-api-xks-ca" pod "fbfb1096-ae10-4197-b046-ae9a310706e8" (UID: "fbfb1096-ae10-4197-b046-ae9a310706e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:23:50.187301 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:50.187253 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-294df\" (UniqueName: \"kubernetes.io/projected/fbfb1096-ae10-4197-b046-ae9a310706e8-kube-api-access-294df\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:50.187301 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:50.187308 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfb1096-ae10-4197-b046-ae9a310706e8-maas-api-xks-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:23:50.849966 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:50.849929 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v" event={"ID":"fbfb1096-ae10-4197-b046-ae9a310706e8","Type":"ContainerDied","Data":"cdde1fdd5b00b9bd8a8594795bfd5dcf92c945c547e39a856f30d207f84ecd10"} Nov 03 22:23:50.850163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:50.849984 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdde1fdd5b00b9bd8a8594795bfd5dcf92c945c547e39a856f30d207f84ecd10" Nov 03 22:23:50.850163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:50.849939 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v" Nov 03 22:23:57.404898 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:57.404857 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5"] Nov 03 22:23:57.405392 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:57.405374 5084 cpu_manager.go:414] "containerMap: removing container" podUID="fbfb1096-ae10-4197-b046-ae9a310706e8" containerName="revoke-keys" Nov 03 22:23:57.405442 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:57.405395 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbfb1096-ae10-4197-b046-ae9a310706e8" containerName="revoke-keys" Nov 03 22:23:57.405505 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:57.405495 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbfb1096-ae10-4197-b046-ae9a310706e8" containerName="revoke-keys" Nov 03 22:23:57.409001 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:57.408976 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5" Nov 03 22:23:57.462904 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:57.462862 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvjvg\" (UniqueName: \"kubernetes.io/projected/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-kube-api-access-xvjvg\") pod \"maas-api-revoke-sub-ae36204d-l59p5\" (UID: \"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5" Nov 03 22:23:57.463110 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:57.462977 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae36204d-l59p5\" (UID: \"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5" Nov 03 22:23:57.463110 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:57.463035 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae36204d-l59p5\" (UID: \"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5" Nov 03 22:23:57.463641 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:57.463617 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5"] Nov 03 22:23:57.510125 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:57.510075 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:23:57.510309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:57.510155 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:23:57.564684 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:57.564640 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvjvg\" (UniqueName: \"kubernetes.io/projected/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-kube-api-access-xvjvg\") pod \"maas-api-revoke-sub-ae36204d-l59p5\" (UID: \"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5" Nov 03 22:23:57.564915 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:57.564805 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae36204d-l59p5\" (UID: \"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5" Nov 03 22:23:57.564915 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:57.564878 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae36204d-l59p5\" (UID: \"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5" Nov 03 22:23:57.565530 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:57.565509 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae36204d-l59p5\" (UID: \"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5" Nov 03 22:23:57.567715 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:57.567688 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae36204d-l59p5\" (UID: \"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5" Nov 03 22:23:57.622674 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:57.622640 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvjvg\" (UniqueName: \"kubernetes.io/projected/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-kube-api-access-xvjvg\") pod \"maas-api-revoke-sub-ae36204d-l59p5\" (UID: \"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5" Nov 03 22:23:57.722979 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:57.722880 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5" Nov 03 22:23:58.102483 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:58.102444 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5"] Nov 03 22:23:58.106126 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:23:58.106095 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef4dfc89_2d3d_4fda_9651_7be3d5cfcb37.slice/crio-3edbcb3839fdd42730160d2608d0b538998bdca71675369ea486c019155969a6 WatchSource:0}: Error finding container 3edbcb3839fdd42730160d2608d0b538998bdca71675369ea486c019155969a6: Status 404 returned error can't find the container with id 3edbcb3839fdd42730160d2608d0b538998bdca71675369ea486c019155969a6 Nov 03 22:23:58.904053 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:58.904013 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5" event={"ID":"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37","Type":"ContainerStarted","Data":"b7b3b0d4429e200ccea51997af027d89a1f6c9e15d59939ef9cf3e2d72da5bc2"} Nov 03 22:23:58.904612 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:58.904098 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5" event={"ID":"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37","Type":"ContainerStarted","Data":"3edbcb3839fdd42730160d2608d0b538998bdca71675369ea486c019155969a6"} Nov 03 22:23:59.023040 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:59.022961 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5" podStartSLOduration=2.022939923 podStartE2EDuration="2.022939923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:59.019780525 +0000 UTC m=+1006.613812513" watchObservedRunningTime="2026-11-03 22:23:59.022939923 +0000 UTC m=+1006.616971896" Nov 03 22:23:59.912595 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:59.912558 5084 generic.go:356] "Generic (PLEG): container finished" podID="ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37" containerID="b7b3b0d4429e200ccea51997af027d89a1f6c9e15d59939ef9cf3e2d72da5bc2" exitCode=0 Nov 03 22:23:59.913030 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:23:59.912600 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5" event={"ID":"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37","Type":"ContainerDied","Data":"b7b3b0d4429e200ccea51997af027d89a1f6c9e15d59939ef9cf3e2d72da5bc2"} Nov 03 22:24:01.085662 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:01.085631 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5" Nov 03 22:24:01.204030 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:01.203943 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-kube-api-access-xvjvg\" (UniqueName: \"kubernetes.io/projected/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-kube-api-access-xvjvg\") pod \"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37\" (UID: \"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37\") " Nov 03 22:24:01.204030 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:01.203978 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-maas-api-openshift-ca\") pod \"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37\" (UID: \"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37\") " Nov 03 22:24:01.204294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:01.204047 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-maas-api-xks-ca\") pod \"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37\" (UID: \"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37\") " Nov 03 22:24:01.204465 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:01.204435 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-maas-api-openshift-ca" pod "ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37" (UID: "ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:24:01.206766 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:01.206729 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-maas-api-xks-ca" pod "ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37" (UID: "ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:24:01.207003 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:01.206969 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-kube-api-access-xvjvg" pod "ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37" (UID: "ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37"). InnerVolumeSpecName "kube-api-access-xvjvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:24:01.305500 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:01.305462 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvjvg\" (UniqueName: \"kubernetes.io/projected/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-kube-api-access-xvjvg\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:01.305500 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:01.305495 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-maas-api-openshift-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:01.305500 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:01.305506 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37-maas-api-xks-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:01.928598 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:01.928567 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5" Nov 03 22:24:01.928818 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:01.928602 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5" event={"ID":"ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37","Type":"ContainerDied","Data":"3edbcb3839fdd42730160d2608d0b538998bdca71675369ea486c019155969a6"} Nov 03 22:24:01.928818 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:01.928653 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3edbcb3839fdd42730160d2608d0b538998bdca71675369ea486c019155969a6" Nov 03 22:24:12.026474 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:24:12.026430 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:24:12.370691 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:12.370604 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8"] Nov 03 22:24:12.370849 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:12.370806 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" podUID="7d44bd1b-06d9-4452-88e8-78fdae1b2203" containerName="istio-proxy" containerID="cri-o://a1e8ad0d89461866f9c9c9eb1a175d80fdf89fe5f6b714f07d1e0e30ad6e63d1" gracePeriod=30 Nov 03 22:24:17.644245 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.644167 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:24:17.707653 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.707615 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-token\" (UniqueName: \"kubernetes.io/projected/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-token\") pod \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " Nov 03 22:24:17.707653 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.707661 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-workload-socket\") pod \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " Nov 03 22:24:17.707893 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.707697 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-podinfo\") pod \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " Nov 03 22:24:17.707893 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.707742 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istiod-ca-cert\") pod \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " Nov 03 22:24:17.707893 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.707769 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-envoy\") pod \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " Nov 03 22:24:17.707893 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.707844 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-workload-certs\") pod \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " Nov 03 22:24:17.708117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.707978 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-credential-socket\") pod \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " Nov 03 22:24:17.708117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.708013 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-data\") pod \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " Nov 03 22:24:17.708117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.708027 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-workload-socket" pod "7d44bd1b-06d9-4452-88e8-78fdae1b2203" (UID: "7d44bd1b-06d9-4452-88e8-78fdae1b2203"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:24:17.708117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.708050 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d44bd1b-06d9-4452-88e8-78fdae1b2203-kube-api-access-vk8s5\" (UniqueName: \"kubernetes.io/projected/7d44bd1b-06d9-4452-88e8-78fdae1b2203-kube-api-access-vk8s5\") pod \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\" (UID: \"7d44bd1b-06d9-4452-88e8-78fdae1b2203\") " Nov 03 22:24:17.708344 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.708275 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-workload-certs" pod "7d44bd1b-06d9-4452-88e8-78fdae1b2203" (UID: "7d44bd1b-06d9-4452-88e8-78fdae1b2203"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:24:17.708344 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.708308 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istiod-ca-cert" pod "7d44bd1b-06d9-4452-88e8-78fdae1b2203" (UID: "7d44bd1b-06d9-4452-88e8-78fdae1b2203"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:24:17.708512 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.708489 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:17.708734 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.708518 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:17.708734 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.708532 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:17.708734 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.708673 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-credential-socket" pod "7d44bd1b-06d9-4452-88e8-78fdae1b2203" (UID: "7d44bd1b-06d9-4452-88e8-78fdae1b2203"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:24:17.708937 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.708875 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-data" pod "7d44bd1b-06d9-4452-88e8-78fdae1b2203" (UID: "7d44bd1b-06d9-4452-88e8-78fdae1b2203"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:24:17.710715 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.710669 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-token" pod "7d44bd1b-06d9-4452-88e8-78fdae1b2203" (UID: "7d44bd1b-06d9-4452-88e8-78fdae1b2203"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:24:17.710882 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.710851 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-podinfo" pod "7d44bd1b-06d9-4452-88e8-78fdae1b2203" (UID: "7d44bd1b-06d9-4452-88e8-78fdae1b2203"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:24:17.710996 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.710931 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-envoy" pod "7d44bd1b-06d9-4452-88e8-78fdae1b2203" (UID: "7d44bd1b-06d9-4452-88e8-78fdae1b2203"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:24:17.711087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.710999 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d44bd1b-06d9-4452-88e8-78fdae1b2203-kube-api-access-vk8s5" pod "7d44bd1b-06d9-4452-88e8-78fdae1b2203" (UID: "7d44bd1b-06d9-4452-88e8-78fdae1b2203"). InnerVolumeSpecName "kube-api-access-vk8s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:24:17.809741 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.809703 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk8s5\" (UniqueName: \"kubernetes.io/projected/7d44bd1b-06d9-4452-88e8-78fdae1b2203-kube-api-access-vk8s5\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:17.809741 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.809738 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:17.809741 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.809757 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:17.809989 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.809767 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:17.809989 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.809777 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:17.809989 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:17.809793 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d44bd1b-06d9-4452-88e8-78fdae1b2203-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:24:18.043609 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:18.043576 5084 generic.go:356] "Generic (PLEG): container finished" podID="7d44bd1b-06d9-4452-88e8-78fdae1b2203" containerID="a1e8ad0d89461866f9c9c9eb1a175d80fdf89fe5f6b714f07d1e0e30ad6e63d1" exitCode=0 Nov 03 22:24:18.043826 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:18.043627 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" Nov 03 22:24:18.043826 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:18.043666 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" event={"ID":"7d44bd1b-06d9-4452-88e8-78fdae1b2203","Type":"ContainerDied","Data":"a1e8ad0d89461866f9c9c9eb1a175d80fdf89fe5f6b714f07d1e0e30ad6e63d1"} Nov 03 22:24:18.043826 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:18.043731 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8" event={"ID":"7d44bd1b-06d9-4452-88e8-78fdae1b2203","Type":"ContainerDied","Data":"faaee4e0ada9d93436e8df11db0918d2d553b02744bbae66afd271f70a1c5abe"} Nov 03 22:24:18.043826 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:18.043764 5084 scope.go:122] "RemoveContainer" containerID="a1e8ad0d89461866f9c9c9eb1a175d80fdf89fe5f6b714f07d1e0e30ad6e63d1" Nov 03 22:24:18.054982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:18.054957 5084 scope.go:122] "RemoveContainer" containerID="a1e8ad0d89461866f9c9c9eb1a175d80fdf89fe5f6b714f07d1e0e30ad6e63d1" Nov 03 22:24:18.055331 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:24:18.055306 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1e8ad0d89461866f9c9c9eb1a175d80fdf89fe5f6b714f07d1e0e30ad6e63d1\": container with ID starting with a1e8ad0d89461866f9c9c9eb1a175d80fdf89fe5f6b714f07d1e0e30ad6e63d1 not found: ID does not exist" containerID="a1e8ad0d89461866f9c9c9eb1a175d80fdf89fe5f6b714f07d1e0e30ad6e63d1" Nov 03 22:24:18.055422 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:18.055341 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1e8ad0d89461866f9c9c9eb1a175d80fdf89fe5f6b714f07d1e0e30ad6e63d1"} err="failed to get container status \"a1e8ad0d89461866f9c9c9eb1a175d80fdf89fe5f6b714f07d1e0e30ad6e63d1\": rpc error: code = NotFound desc = could not find container \"a1e8ad0d89461866f9c9c9eb1a175d80fdf89fe5f6b714f07d1e0e30ad6e63d1\": container with ID starting with a1e8ad0d89461866f9c9c9eb1a175d80fdf89fe5f6b714f07d1e0e30ad6e63d1 not found: ID does not exist" Nov 03 22:24:18.213872 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:18.213829 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8"] Nov 03 22:24:18.217927 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:18.217885 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-sj8g8"] Nov 03 22:24:18.633108 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:18.632465 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d44bd1b-06d9-4452-88e8-78fdae1b2203" path="/var/lib/kubelet/pods/7d44bd1b-06d9-4452-88e8-78fdae1b2203/volumes" Nov 03 22:24:20.212830 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.212786 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp"] Nov 03 22:24:20.213606 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.213578 5084 cpu_manager.go:414] "containerMap: removing container" podUID="ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37" containerName="revoke-keys" Nov 03 22:24:20.213606 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.213603 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37" containerName="revoke-keys" Nov 03 22:24:20.213782 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.213661 5084 cpu_manager.go:414] "containerMap: removing container" podUID="7d44bd1b-06d9-4452-88e8-78fdae1b2203" containerName="istio-proxy" Nov 03 22:24:20.213782 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.213671 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d44bd1b-06d9-4452-88e8-78fdae1b2203" containerName="istio-proxy" Nov 03 22:24:20.213943 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.213923 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37" containerName="revoke-keys" Nov 03 22:24:20.214047 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.213966 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d44bd1b-06d9-4452-88e8-78fdae1b2203" containerName="istio-proxy" Nov 03 22:24:20.219193 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.219165 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.233558 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.233528 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.233728 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.233567 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.233728 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.233593 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487f04a7-abb9-40e2-9744-7068195f5e00-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.233728 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.233656 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md55q\" (UniqueName: \"kubernetes.io/projected/487f04a7-abb9-40e2-9744-7068195f5e00-kube-api-access-md55q\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.233914 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.233780 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.233972 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.233912 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.233972 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.233962 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487f04a7-abb9-40e2-9744-7068195f5e00-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.234092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.234045 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487f04a7-abb9-40e2-9744-7068195f5e00-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.234165 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.234148 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.306407 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.306371 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp"] Nov 03 22:24:20.335513 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.335477 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.335720 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.335521 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.335720 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.335566 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487f04a7-abb9-40e2-9744-7068195f5e00-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.335720 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.335592 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md55q\" (UniqueName: \"kubernetes.io/projected/487f04a7-abb9-40e2-9744-7068195f5e00-kube-api-access-md55q\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.335720 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.335623 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.335720 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.335706 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.335973 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.335760 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487f04a7-abb9-40e2-9744-7068195f5e00-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.335973 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.335816 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487f04a7-abb9-40e2-9744-7068195f5e00-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.335973 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.335852 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.335973 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.335966 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.336199 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.336013 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.336253 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.336206 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.336617 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.336566 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.336748 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.336714 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487f04a7-abb9-40e2-9744-7068195f5e00-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.338377 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.338353 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487f04a7-abb9-40e2-9744-7068195f5e00-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.338702 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.338685 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.393956 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.393913 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487f04a7-abb9-40e2-9744-7068195f5e00-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.394180 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.393927 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md55q\" (UniqueName: \"kubernetes.io/projected/487f04a7-abb9-40e2-9744-7068195f5e00-kube-api-access-md55q\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.533411 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.533370 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:20.967656 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.967627 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp"] Nov 03 22:24:20.969469 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:24:20.969436 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487f04a7_abb9_40e2_9744_7068195f5e00.slice/crio-b081394fb0ad758e5468a88e80d8b8bbdf8452ba598fcd17cd5cb74d4dce4e0e WatchSource:0}: Error finding container b081394fb0ad758e5468a88e80d8b8bbdf8452ba598fcd17cd5cb74d4dce4e0e: Status 404 returned error can't find the container with id b081394fb0ad758e5468a88e80d8b8bbdf8452ba598fcd17cd5cb74d4dce4e0e Nov 03 22:24:20.971828 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.971791 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:24:20.971931 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.971873 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:24:20.971931 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:20.971909 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:24:21.074668 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:21.074621 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" event={"ID":"487f04a7-abb9-40e2-9744-7068195f5e00","Type":"ContainerStarted","Data":"19c33cfca9ed5d86c52f78b598c2ac06301559c652e23397ae53c5ef8e327ba1"} Nov 03 22:24:21.074873 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:21.074694 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" event={"ID":"487f04a7-abb9-40e2-9744-7068195f5e00","Type":"ContainerStarted","Data":"b081394fb0ad758e5468a88e80d8b8bbdf8452ba598fcd17cd5cb74d4dce4e0e"} Nov 03 22:24:21.196076 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:21.195719 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" podStartSLOduration=1.195700249 podStartE2EDuration="1.195700249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:24:21.191958392 +0000 UTC m=+1028.785990366" watchObservedRunningTime="2026-11-03 22:24:21.195700249 +0000 UTC m=+1028.789732224" Nov 03 22:24:21.533706 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:21.533665 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:22.535881 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:22.535785 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:24:22.535881 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:22.535848 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" podUID="487f04a7-abb9-40e2-9744-7068195f5e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 22:24:23.534520 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:23.534479 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:24:23.534715 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:23.534536 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" podUID="487f04a7-abb9-40e2-9744-7068195f5e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 22:24:24.534363 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:24.534295 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:24:24.534363 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:24.534353 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" podUID="487f04a7-abb9-40e2-9744-7068195f5e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 22:24:25.534582 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:25.534540 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:24:25.535001 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:25.534612 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" podUID="487f04a7-abb9-40e2-9744-7068195f5e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 22:24:26.534574 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:26.534536 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:24:26.534767 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:26.534605 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" podUID="487f04a7-abb9-40e2-9744-7068195f5e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 22:24:27.534563 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:27.534520 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 22:24:27.534757 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:27.534590 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" podUID="487f04a7-abb9-40e2-9744-7068195f5e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 22:24:28.538959 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:28.538924 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:28.539509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:28.539306 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:28.540030 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:28.540008 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:24:28.909102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:28.908987 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn"] Nov 03 22:24:28.913182 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:28.913154 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:28.955856 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:28.955802 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-tmp-dir\") pod \"distinct-w0-f61714-kserve-7655b788d8-p8kqn\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:28.956108 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:28.955989 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b28118-7e39-4b66-be5f-f5a61a199f85-tls-certs\") pod \"distinct-w0-f61714-kserve-7655b788d8-p8kqn\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:28.956186 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:28.956152 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-dshm\") pod \"distinct-w0-f61714-kserve-7655b788d8-p8kqn\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:28.956252 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:28.956188 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-home\") pod \"distinct-w0-f61714-kserve-7655b788d8-p8kqn\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:28.956294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:28.956252 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-model-cache\") pod \"distinct-w0-f61714-kserve-7655b788d8-p8kqn\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:28.956294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:28.956286 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8t6t\" (UniqueName: \"kubernetes.io/projected/a8b28118-7e39-4b66-be5f-f5a61a199f85-kube-api-access-v8t6t\") pod \"distinct-w0-f61714-kserve-7655b788d8-p8kqn\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:28.966838 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:28.966798 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"distinct-w0-f61714-kserve-self-signed-certs\"" Nov 03 22:24:28.966838 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:28.966825 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"openshift-service-ca.crt\"" Nov 03 22:24:28.967136 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:28.966925 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"kube-root-ca.crt\"" Nov 03 22:24:28.967136 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:28.967045 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f61714\"/\"default-dockercfg-hrt5c\"" Nov 03 22:24:28.967593 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:28.967568 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn"] Nov 03 22:24:29.057040 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:29.056999 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-model-cache\") pod \"distinct-w0-f61714-kserve-7655b788d8-p8kqn\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:29.057264 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:29.057052 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8t6t\" (UniqueName: \"kubernetes.io/projected/a8b28118-7e39-4b66-be5f-f5a61a199f85-kube-api-access-v8t6t\") pod \"distinct-w0-f61714-kserve-7655b788d8-p8kqn\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:29.057264 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:29.057148 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-tmp-dir\") pod \"distinct-w0-f61714-kserve-7655b788d8-p8kqn\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:29.075621 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:29.075173 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b28118-7e39-4b66-be5f-f5a61a199f85-tls-certs\") pod \"distinct-w0-f61714-kserve-7655b788d8-p8kqn\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:29.075621 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:29.075329 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-dshm\") pod \"distinct-w0-f61714-kserve-7655b788d8-p8kqn\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:29.075621 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:29.075356 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-home\") pod \"distinct-w0-f61714-kserve-7655b788d8-p8kqn\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:29.075898 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:29.075862 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-home\") pod \"distinct-w0-f61714-kserve-7655b788d8-p8kqn\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:29.076233 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:29.076206 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-model-cache\") pod \"distinct-w0-f61714-kserve-7655b788d8-p8kqn\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:29.076418 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:29.076288 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-tmp-dir\") pod \"distinct-w0-f61714-kserve-7655b788d8-p8kqn\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:29.080498 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:29.080464 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b28118-7e39-4b66-be5f-f5a61a199f85-tls-certs\") pod \"distinct-w0-f61714-kserve-7655b788d8-p8kqn\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:29.093790 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:29.091693 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-dshm\") pod \"distinct-w0-f61714-kserve-7655b788d8-p8kqn\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:29.135159 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:29.135127 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8t6t\" (UniqueName: \"kubernetes.io/projected/a8b28118-7e39-4b66-be5f-f5a61a199f85-kube-api-access-v8t6t\") pod \"distinct-w0-f61714-kserve-7655b788d8-p8kqn\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:29.227261 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:29.227161 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:29.691816 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:29.691767 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn"] Nov 03 22:24:30.142196 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:30.142143 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" event={"ID":"a8b28118-7e39-4b66-be5f-f5a61a199f85","Type":"ContainerStarted","Data":"063b182003e3bb3a31fb2180b042ff1921530c892a199257dea6b73e4e3124b7"} Nov 03 22:24:30.142196 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:30.142195 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" event={"ID":"a8b28118-7e39-4b66-be5f-f5a61a199f85","Type":"ContainerStarted","Data":"fb3f8e14d3036b97b0d5096c9d6cd1421d383803cf5d4f8ade17572a2a41d898"} Nov 03 22:24:30.227825 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:30.227778 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:30.260995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:30.260926 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podStartSLOduration=2.26090965 podStartE2EDuration="2.26090965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:24:30.259006123 +0000 UTC m=+1037.853038098" watchObservedRunningTime="2026-11-03 22:24:30.26090965 +0000 UTC m=+1037.854941602" Nov 03 22:24:35.686874 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.686824 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r"] Nov 03 22:24:35.691244 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.691214 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.757574 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.757533 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r"] Nov 03 22:24:35.769394 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.769356 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-token\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.769556 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.769411 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c395459-9f53-4792-9b3a-7a3a1f733a65-istiod-ca-cert\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.769556 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.769517 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-data\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.769631 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.769612 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-podinfo\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.769715 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.769645 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zsm7\" (UniqueName: \"kubernetes.io/projected/0c395459-9f53-4792-9b3a-7a3a1f733a65-kube-api-access-9zsm7\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.769759 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.769705 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-workload-certs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.769859 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.769840 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-envoy\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.769932 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.769918 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-workload-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.769965 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.769940 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-credential-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.793567 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.793532 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f7e6db-openshift-default-dockercfg-dfr7z\"" Nov 03 22:24:35.871479 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.871437 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-workload-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.871479 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.871479 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-credential-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.871743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.871662 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-token\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.871743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.871722 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c395459-9f53-4792-9b3a-7a3a1f733a65-istiod-ca-cert\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.871851 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.871763 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-data\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.871851 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.871816 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-podinfo\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.871950 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.871848 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-credential-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.871950 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.871851 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zsm7\" (UniqueName: \"kubernetes.io/projected/0c395459-9f53-4792-9b3a-7a3a1f733a65-kube-api-access-9zsm7\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.871950 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.871899 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-workload-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.871950 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.871915 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-workload-certs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.872183 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.872027 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-envoy\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.872344 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.872313 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-data\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.872472 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.872444 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-workload-certs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.872779 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.872759 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c395459-9f53-4792-9b3a-7a3a1f733a65-istiod-ca-cert\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.874380 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.874356 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-envoy\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.874854 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.874832 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-podinfo\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.931235 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.931200 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-token\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:35.931540 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:35.931511 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zsm7\" (UniqueName: \"kubernetes.io/projected/0c395459-9f53-4792-9b3a-7a3a1f733a65-kube-api-access-9zsm7\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:36.006039 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:36.005931 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:36.453927 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:36.453885 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r"] Nov 03 22:24:36.490907 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:24:36.490876 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c395459_9f53_4792_9b3a_7a3a1f733a65.slice/crio-7aafe600b2f00fb5952814f60a9c6863834cd5b9f09a0cc5e05f73a459e49712 WatchSource:0}: Error finding container 7aafe600b2f00fb5952814f60a9c6863834cd5b9f09a0cc5e05f73a459e49712: Status 404 returned error can't find the container with id 7aafe600b2f00fb5952814f60a9c6863834cd5b9f09a0cc5e05f73a459e49712 Nov 03 22:24:36.493419 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:36.493385 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:24:36.493512 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:36.493460 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:24:36.493512 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:36.493489 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:24:37.195584 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:37.195550 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" event={"ID":"0c395459-9f53-4792-9b3a-7a3a1f733a65","Type":"ContainerStarted","Data":"0419033b12c66dd1b10445fe72a2da15fbfc2cd65dad44bc23f86000ff746db2"} Nov 03 22:24:37.196154 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:37.195602 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" event={"ID":"0c395459-9f53-4792-9b3a-7a3a1f733a65","Type":"ContainerStarted","Data":"7aafe600b2f00fb5952814f60a9c6863834cd5b9f09a0cc5e05f73a459e49712"} Nov 03 22:24:37.316994 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:37.316929 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" podStartSLOduration=2.316910733 podStartE2EDuration="2.316910733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:24:37.315248883 +0000 UTC m=+1044.909280851" watchObservedRunningTime="2026-11-03 22:24:37.316910733 +0000 UTC m=+1044.910942707" Nov 03 22:24:38.007222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:38.007142 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:38.008691 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:38.008401 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:24:38.008691 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:38.008463 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" podUID="0c395459-9f53-4792-9b3a-7a3a1f733a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 22:24:39.007340 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:39.007301 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:24:39.007760 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:39.007385 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" podUID="0c395459-9f53-4792-9b3a-7a3a1f733a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 22:24:39.227781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:39.227733 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:39.241195 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:39.241161 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:40.006976 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:40.006935 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:24:40.007208 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:40.006999 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" podUID="0c395459-9f53-4792-9b3a-7a3a1f733a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 22:24:40.236941 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:40.236909 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:24:41.006791 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:41.006749 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:24:41.007179 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:41.007140 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" podUID="0c395459-9f53-4792-9b3a-7a3a1f733a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 22:24:42.006313 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:42.006270 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:24:42.008780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:42.006336 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" podUID="0c395459-9f53-4792-9b3a-7a3a1f733a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 22:24:42.024183 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:24:42.024133 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:24:43.006821 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:43.006771 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:24:43.007365 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:43.006850 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" podUID="0c395459-9f53-4792-9b3a-7a3a1f733a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 22:24:43.902221 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:43.902178 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-hhkg4"] Nov 03 22:24:43.906385 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:43.906360 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-hhkg4" Nov 03 22:24:43.959992 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:43.959941 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-hhkg4"] Nov 03 22:24:43.971425 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:43.971377 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pspl6\" (UniqueName: \"kubernetes.io/projected/5117984d-dade-4802-a318-948f508c7658-kube-api-access-pspl6\") pod \"odh-model-controller-769875db56-hhkg4\" (UID: \"5117984d-dade-4802-a318-948f508c7658\") " pod="opendatahub/odh-model-controller-769875db56-hhkg4" Nov 03 22:24:43.971645 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:43.971534 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5117984d-dade-4802-a318-948f508c7658-metrics-certs\") pod \"odh-model-controller-769875db56-hhkg4\" (UID: \"5117984d-dade-4802-a318-948f508c7658\") " pod="opendatahub/odh-model-controller-769875db56-hhkg4" Nov 03 22:24:43.971764 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:43.971714 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5117984d-dade-4802-a318-948f508c7658-cert\") pod \"odh-model-controller-769875db56-hhkg4\" (UID: \"5117984d-dade-4802-a318-948f508c7658\") " pod="opendatahub/odh-model-controller-769875db56-hhkg4" Nov 03 22:24:44.006777 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:44.006734 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 22:24:44.010994 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:44.010954 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-48bmq\"" Nov 03 22:24:44.011497 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:44.011011 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:44.011497 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:44.011049 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 22:24:44.011497 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:44.011291 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:44.012175 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:44.012156 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:24:44.072753 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:44.072706 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5117984d-dade-4802-a318-948f508c7658-metrics-certs\") pod \"odh-model-controller-769875db56-hhkg4\" (UID: \"5117984d-dade-4802-a318-948f508c7658\") " pod="opendatahub/odh-model-controller-769875db56-hhkg4" Nov 03 22:24:44.072947 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:44.072851 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5117984d-dade-4802-a318-948f508c7658-cert\") pod \"odh-model-controller-769875db56-hhkg4\" (UID: \"5117984d-dade-4802-a318-948f508c7658\") " pod="opendatahub/odh-model-controller-769875db56-hhkg4" Nov 03 22:24:44.072947 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:44.072924 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pspl6\" (UniqueName: \"kubernetes.io/projected/5117984d-dade-4802-a318-948f508c7658-kube-api-access-pspl6\") pod \"odh-model-controller-769875db56-hhkg4\" (UID: \"5117984d-dade-4802-a318-948f508c7658\") " pod="opendatahub/odh-model-controller-769875db56-hhkg4" Nov 03 22:24:44.075877 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:44.075836 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5117984d-dade-4802-a318-948f508c7658-metrics-certs\") pod \"odh-model-controller-769875db56-hhkg4\" (UID: \"5117984d-dade-4802-a318-948f508c7658\") " pod="opendatahub/odh-model-controller-769875db56-hhkg4" Nov 03 22:24:44.075877 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:44.075858 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5117984d-dade-4802-a318-948f508c7658-cert\") pod \"odh-model-controller-769875db56-hhkg4\" (UID: \"5117984d-dade-4802-a318-948f508c7658\") " pod="opendatahub/odh-model-controller-769875db56-hhkg4" Nov 03 22:24:44.131597 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:44.131564 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pspl6\" (UniqueName: \"kubernetes.io/projected/5117984d-dade-4802-a318-948f508c7658-kube-api-access-pspl6\") pod \"odh-model-controller-769875db56-hhkg4\" (UID: \"5117984d-dade-4802-a318-948f508c7658\") " pod="opendatahub/odh-model-controller-769875db56-hhkg4" Nov 03 22:24:44.221315 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:44.221213 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-hhkg4" Nov 03 22:24:44.611618 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:24:44.611580 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5117984d_dade_4802_a318_948f508c7658.slice/crio-2242a99c0c60260e8ed416a9472b424b1c23fc7878f5b31215919d58f4cf023c WatchSource:0}: Error finding container 2242a99c0c60260e8ed416a9472b424b1c23fc7878f5b31215919d58f4cf023c: Status 404 returned error can't find the container with id 2242a99c0c60260e8ed416a9472b424b1c23fc7878f5b31215919d58f4cf023c Nov 03 22:24:44.611775 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:44.611746 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-hhkg4"] Nov 03 22:24:45.263261 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:45.263226 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-hhkg4" event={"ID":"5117984d-dade-4802-a318-948f508c7658","Type":"ContainerStarted","Data":"2242a99c0c60260e8ed416a9472b424b1c23fc7878f5b31215919d58f4cf023c"} Nov 03 22:24:48.293286 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:48.293241 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-hhkg4" event={"ID":"5117984d-dade-4802-a318-948f508c7658","Type":"ContainerStarted","Data":"469c25aa77db6e94a622f046e2f47be68f54cbb8da0f59235ed8295d26d9baee"} Nov 03 22:24:48.293787 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:48.293337 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-hhkg4" Nov 03 22:24:48.411739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:48.411626 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-hhkg4" podStartSLOduration=1.9677779389999999 podStartE2EDuration="5.411608574s" totalImagesPullingTime="3.443830635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:24:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:24:48.4097623 +0000 UTC m=+1056.003794278" watchObservedRunningTime="2026-11-03 22:24:48.411608574 +0000 UTC m=+1056.005640553" Nov 03 22:24:59.304697 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:24:59.304660 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-hhkg4" Nov 03 22:25:12.023790 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:25:12.023749 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:25:12.378193 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:12.378090 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44"] Nov 03 22:25:12.378393 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:12.378364 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" podUID="e541ea68-f573-400b-b0ef-9383d6bcba51" containerName="istio-proxy" containerID="cri-o://fae06f5b57cb11d768172967c4ab7d0d08b3cce62303ec4e5e74a623342606be" gracePeriod=30 Nov 03 22:25:17.509955 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:17.509918 5084 generic.go:356] "Generic (PLEG): container finished" podID="e541ea68-f573-400b-b0ef-9383d6bcba51" containerID="fae06f5b57cb11d768172967c4ab7d0d08b3cce62303ec4e5e74a623342606be" exitCode=0 Nov 03 22:25:17.510386 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:17.509990 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" event={"ID":"e541ea68-f573-400b-b0ef-9383d6bcba51","Type":"ContainerDied","Data":"fae06f5b57cb11d768172967c4ab7d0d08b3cce62303ec4e5e74a623342606be"} Nov 03 22:25:18.074419 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.074381 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:25:18.166495 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.166381 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e541ea68-f573-400b-b0ef-9383d6bcba51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e541ea68-f573-400b-b0ef-9383d6bcba51-istiod-ca-cert\") pod \"e541ea68-f573-400b-b0ef-9383d6bcba51\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " Nov 03 22:25:18.166495 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.166477 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-credential-socket\") pod \"e541ea68-f573-400b-b0ef-9383d6bcba51\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " Nov 03 22:25:18.166806 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.166548 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-data\") pod \"e541ea68-f573-400b-b0ef-9383d6bcba51\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " Nov 03 22:25:18.166806 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.166583 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-podinfo\") pod \"e541ea68-f573-400b-b0ef-9383d6bcba51\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " Nov 03 22:25:18.166806 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.166630 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-workload-certs\") pod \"e541ea68-f573-400b-b0ef-9383d6bcba51\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " Nov 03 22:25:18.166806 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.166676 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-token\" (UniqueName: \"kubernetes.io/projected/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-token\") pod \"e541ea68-f573-400b-b0ef-9383d6bcba51\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " Nov 03 22:25:18.166806 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.166712 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-envoy\") pod \"e541ea68-f573-400b-b0ef-9383d6bcba51\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " Nov 03 22:25:18.166806 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.166779 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e541ea68-f573-400b-b0ef-9383d6bcba51-kube-api-access-pkq4w\" (UniqueName: \"kubernetes.io/projected/e541ea68-f573-400b-b0ef-9383d6bcba51-kube-api-access-pkq4w\") pod \"e541ea68-f573-400b-b0ef-9383d6bcba51\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " Nov 03 22:25:18.167126 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.166812 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-workload-socket\") pod \"e541ea68-f573-400b-b0ef-9383d6bcba51\" (UID: \"e541ea68-f573-400b-b0ef-9383d6bcba51\") " Nov 03 22:25:18.167126 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.166880 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e541ea68-f573-400b-b0ef-9383d6bcba51-istiod-ca-cert" pod "e541ea68-f573-400b-b0ef-9383d6bcba51" (UID: "e541ea68-f573-400b-b0ef-9383d6bcba51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:18.167240 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.167142 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-workload-certs" pod "e541ea68-f573-400b-b0ef-9383d6bcba51" (UID: "e541ea68-f573-400b-b0ef-9383d6bcba51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:25:18.167240 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.167211 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-credential-socket" pod "e541ea68-f573-400b-b0ef-9383d6bcba51" (UID: "e541ea68-f573-400b-b0ef-9383d6bcba51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:25:18.167370 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.167350 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e541ea68-f573-400b-b0ef-9383d6bcba51-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:25:18.167432 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.167375 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:25:18.167432 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.167391 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:25:18.167552 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.167434 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-workload-socket" pod "e541ea68-f573-400b-b0ef-9383d6bcba51" (UID: "e541ea68-f573-400b-b0ef-9383d6bcba51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:25:18.167552 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.167464 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-data" pod "e541ea68-f573-400b-b0ef-9383d6bcba51" (UID: "e541ea68-f573-400b-b0ef-9383d6bcba51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:25:18.169361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.169327 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-podinfo" pod "e541ea68-f573-400b-b0ef-9383d6bcba51" (UID: "e541ea68-f573-400b-b0ef-9383d6bcba51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:25:18.169535 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.169509 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-envoy" pod "e541ea68-f573-400b-b0ef-9383d6bcba51" (UID: "e541ea68-f573-400b-b0ef-9383d6bcba51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:25:18.169713 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.169686 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e541ea68-f573-400b-b0ef-9383d6bcba51-kube-api-access-pkq4w" pod "e541ea68-f573-400b-b0ef-9383d6bcba51" (UID: "e541ea68-f573-400b-b0ef-9383d6bcba51"). InnerVolumeSpecName "kube-api-access-pkq4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:18.169851 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.169781 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-token" pod "e541ea68-f573-400b-b0ef-9383d6bcba51" (UID: "e541ea68-f573-400b-b0ef-9383d6bcba51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:18.269047 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.269003 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:25:18.269047 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.269048 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:25:18.269322 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.269079 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:25:18.269322 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.269094 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:25:18.269322 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.269108 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkq4w\" (UniqueName: \"kubernetes.io/projected/e541ea68-f573-400b-b0ef-9383d6bcba51-kube-api-access-pkq4w\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:25:18.269322 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.269124 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541ea68-f573-400b-b0ef-9383d6bcba51-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:25:18.517893 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.517859 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" event={"ID":"e541ea68-f573-400b-b0ef-9383d6bcba51","Type":"ContainerDied","Data":"83571fb90b798f8fea4c54feee09d9664c3a86108f9e6f3ee031ae45b7309beb"} Nov 03 22:25:18.517893 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.517891 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44" Nov 03 22:25:18.518433 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.517953 5084 scope.go:122] "RemoveContainer" containerID="fae06f5b57cb11d768172967c4ab7d0d08b3cce62303ec4e5e74a623342606be" Nov 03 22:25:18.710780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.710743 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44"] Nov 03 22:25:18.715938 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:18.715905 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-k2q44"] Nov 03 22:25:20.627707 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.627672 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e541ea68-f573-400b-b0ef-9383d6bcba51" path="/var/lib/kubelet/pods/e541ea68-f573-400b-b0ef-9383d6bcba51/volumes" Nov 03 22:25:20.721128 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.721056 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq"] Nov 03 22:25:20.721660 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.721644 5084 cpu_manager.go:414] "containerMap: removing container" podUID="e541ea68-f573-400b-b0ef-9383d6bcba51" containerName="istio-proxy" Nov 03 22:25:20.721712 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.721662 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e541ea68-f573-400b-b0ef-9383d6bcba51" containerName="istio-proxy" Nov 03 22:25:20.721762 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.721744 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="e541ea68-f573-400b-b0ef-9383d6bcba51" containerName="istio-proxy" Nov 03 22:25:20.727860 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.727833 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:20.779959 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.779920 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq"] Nov 03 22:25:20.796051 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.796003 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-home\") pod \"distinct-w4-f7e6db-kserve-74db565c5-8hnwq\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:20.796275 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.796077 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-model-cache\") pod \"distinct-w4-f7e6db-kserve-74db565c5-8hnwq\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:20.796275 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.796213 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-dshm\") pod \"distinct-w4-f7e6db-kserve-74db565c5-8hnwq\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:20.796431 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.796283 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvxgq\" (UniqueName: \"kubernetes.io/projected/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-kube-api-access-vvxgq\") pod \"distinct-w4-f7e6db-kserve-74db565c5-8hnwq\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:20.796431 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.796371 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-tls-certs\") pod \"distinct-w4-f7e6db-kserve-74db565c5-8hnwq\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:20.796431 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.796416 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-tmp-dir\") pod \"distinct-w4-f7e6db-kserve-74db565c5-8hnwq\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:20.826982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.826937 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f7e6db\"/\"openshift-service-ca.crt\"" Nov 03 22:25:20.826982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.826966 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f7e6db\"/\"distinct-w4-f7e6db-kserve-self-signed-certs\"" Nov 03 22:25:20.832271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.832242 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f7e6db\"/\"kube-root-ca.crt\"" Nov 03 22:25:20.832271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.832268 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f7e6db\"/\"default-dockercfg-xdf7f\"" Nov 03 22:25:20.916632 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.916539 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-tmp-dir\") pod \"distinct-w4-f7e6db-kserve-74db565c5-8hnwq\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:20.916632 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.916621 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-home\") pod \"distinct-w4-f7e6db-kserve-74db565c5-8hnwq\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:20.916882 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.916647 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-model-cache\") pod \"distinct-w4-f7e6db-kserve-74db565c5-8hnwq\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:20.916882 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.916697 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-dshm\") pod \"distinct-w4-f7e6db-kserve-74db565c5-8hnwq\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:20.916882 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.916733 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvxgq\" (UniqueName: \"kubernetes.io/projected/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-kube-api-access-vvxgq\") pod \"distinct-w4-f7e6db-kserve-74db565c5-8hnwq\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:20.916882 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.916809 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-tls-certs\") pod \"distinct-w4-f7e6db-kserve-74db565c5-8hnwq\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:20.917102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.917055 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-tmp-dir\") pod \"distinct-w4-f7e6db-kserve-74db565c5-8hnwq\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:20.917155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.917096 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-home\") pod \"distinct-w4-f7e6db-kserve-74db565c5-8hnwq\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:20.917368 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.917340 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-model-cache\") pod \"distinct-w4-f7e6db-kserve-74db565c5-8hnwq\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:20.919248 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.919218 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-dshm\") pod \"distinct-w4-f7e6db-kserve-74db565c5-8hnwq\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:20.919411 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.919394 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-tls-certs\") pod \"distinct-w4-f7e6db-kserve-74db565c5-8hnwq\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:20.974773 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:20.974732 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvxgq\" (UniqueName: \"kubernetes.io/projected/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-kube-api-access-vvxgq\") pod \"distinct-w4-f7e6db-kserve-74db565c5-8hnwq\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:21.040889 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:21.040845 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:21.488318 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:21.488284 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq"] Nov 03 22:25:21.490342 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:25:21.490312 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca63bdeb_70de_46b0_b8ec_ba3e5a57fa49.slice/crio-6afbfcd33dbd98326a1d5fea5b959548ddbf4bc1bb326889ff70237e176d2ad3 WatchSource:0}: Error finding container 6afbfcd33dbd98326a1d5fea5b959548ddbf4bc1bb326889ff70237e176d2ad3: Status 404 returned error can't find the container with id 6afbfcd33dbd98326a1d5fea5b959548ddbf4bc1bb326889ff70237e176d2ad3 Nov 03 22:25:21.543812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:21.543770 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" event={"ID":"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49","Type":"ContainerStarted","Data":"6afbfcd33dbd98326a1d5fea5b959548ddbf4bc1bb326889ff70237e176d2ad3"} Nov 03 22:25:22.553276 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:22.553233 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" event={"ID":"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49","Type":"ContainerStarted","Data":"0aca803caaefa0066cc726e6bce7b3a460febd46ef87037b3f01ea39d297c1ed"} Nov 03 22:25:22.669550 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:22.669492 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podStartSLOduration=2.669478137 podStartE2EDuration="2.669478137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:22.668418817 +0000 UTC m=+1090.262450791" watchObservedRunningTime="2026-11-03 22:25:22.669478137 +0000 UTC m=+1090.263510111" Nov 03 22:25:23.042037 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:23.041998 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:31.041625 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:31.041575 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:31.055899 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:31.055859 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:31.623855 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:25:31.623824 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:25:42.023284 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:25:42.023235 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:26:06.331624 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.331587 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5"] Nov 03 22:26:06.335754 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.335731 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.391669 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.391629 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5"] Nov 03 22:26:06.436624 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.436590 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"default-dockercfg-9jmx4\"" Nov 03 22:26:06.440293 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.440250 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3927af-60a2-4432-99d6-b999a96f315f-tls-certs\") pod \"distinct-w1-3378fd-kserve-77b88d77dc-2jxr5\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.440517 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.440305 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-model-cache\") pod \"distinct-w1-3378fd-kserve-77b88d77dc-2jxr5\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.440517 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.440495 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s2hj\" (UniqueName: \"kubernetes.io/projected/3b3927af-60a2-4432-99d6-b999a96f315f-kube-api-access-7s2hj\") pod \"distinct-w1-3378fd-kserve-77b88d77dc-2jxr5\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.440620 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.440551 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-tmp-dir\") pod \"distinct-w1-3378fd-kserve-77b88d77dc-2jxr5\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.440674 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.440650 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-dshm\") pod \"distinct-w1-3378fd-kserve-77b88d77dc-2jxr5\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.440715 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.440694 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-home\") pod \"distinct-w1-3378fd-kserve-77b88d77dc-2jxr5\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.443324 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.443289 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"openshift-service-ca.crt\"" Nov 03 22:26:06.443324 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.443317 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"kube-root-ca.crt\"" Nov 03 22:26:06.443495 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.443325 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3378fd\"/\"distinct-w1-3378fd-kserve-self-signed-certs\"" Nov 03 22:26:06.542001 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.541953 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-dshm\") pod \"distinct-w1-3378fd-kserve-77b88d77dc-2jxr5\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.542230 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.542010 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-home\") pod \"distinct-w1-3378fd-kserve-77b88d77dc-2jxr5\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.542230 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.542134 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3927af-60a2-4432-99d6-b999a96f315f-tls-certs\") pod \"distinct-w1-3378fd-kserve-77b88d77dc-2jxr5\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.542230 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.542163 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-model-cache\") pod \"distinct-w1-3378fd-kserve-77b88d77dc-2jxr5\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.542661 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.542631 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-model-cache\") pod \"distinct-w1-3378fd-kserve-77b88d77dc-2jxr5\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.542963 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.542934 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s2hj\" (UniqueName: \"kubernetes.io/projected/3b3927af-60a2-4432-99d6-b999a96f315f-kube-api-access-7s2hj\") pod \"distinct-w1-3378fd-kserve-77b88d77dc-2jxr5\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.543119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.542978 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-tmp-dir\") pod \"distinct-w1-3378fd-kserve-77b88d77dc-2jxr5\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.543119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.543047 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-home\") pod \"distinct-w1-3378fd-kserve-77b88d77dc-2jxr5\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.543380 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.543359 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-tmp-dir\") pod \"distinct-w1-3378fd-kserve-77b88d77dc-2jxr5\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.545179 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.545151 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-dshm\") pod \"distinct-w1-3378fd-kserve-77b88d77dc-2jxr5\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.545467 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.545443 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3927af-60a2-4432-99d6-b999a96f315f-tls-certs\") pod \"distinct-w1-3378fd-kserve-77b88d77dc-2jxr5\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.605791 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.605658 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s2hj\" (UniqueName: \"kubernetes.io/projected/3b3927af-60a2-4432-99d6-b999a96f315f-kube-api-access-7s2hj\") pod \"distinct-w1-3378fd-kserve-77b88d77dc-2jxr5\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.650545 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.650500 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:06.932855 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.932758 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-95nwr-7rhld"] Nov 03 22:26:06.938691 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.938663 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-95nwr-7rhld" Nov 03 22:26:06.989643 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.989605 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-95nwr-7rhld"] Nov 03 22:26:06.992217 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:06.992192 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cqv8d\"" Nov 03 22:26:07.051352 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.051310 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxzxd\" (UniqueName: \"kubernetes.io/projected/2f0612fd-99a8-4b86-8345-69dc450d4b75-kube-api-access-bxzxd\") pod \"periodic-gathering-95nwr-7rhld\" (UID: \"2f0612fd-99a8-4b86-8345-69dc450d4b75\") " pod="openshift-insights/periodic-gathering-95nwr-7rhld" Nov 03 22:26:07.051550 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.051479 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2f0612fd-99a8-4b86-8345-69dc450d4b75-archives-path\") pod \"periodic-gathering-95nwr-7rhld\" (UID: \"2f0612fd-99a8-4b86-8345-69dc450d4b75\") " pod="openshift-insights/periodic-gathering-95nwr-7rhld" Nov 03 22:26:07.051550 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.051519 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f0612fd-99a8-4b86-8345-69dc450d4b75-service-ca-bundle\") pod \"periodic-gathering-95nwr-7rhld\" (UID: \"2f0612fd-99a8-4b86-8345-69dc450d4b75\") " pod="openshift-insights/periodic-gathering-95nwr-7rhld" Nov 03 22:26:07.051550 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.051547 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f0612fd-99a8-4b86-8345-69dc450d4b75-trusted-ca-bundle\") pod \"periodic-gathering-95nwr-7rhld\" (UID: \"2f0612fd-99a8-4b86-8345-69dc450d4b75\") " pod="openshift-insights/periodic-gathering-95nwr-7rhld" Nov 03 22:26:07.102662 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.102627 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5"] Nov 03 22:26:07.104291 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:26:07.104260 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b3927af_60a2_4432_99d6_b999a96f315f.slice/crio-bb7bd871465d856415dd0b976be6d0f13aac5e1d4f726640e0788fab982cae20 WatchSource:0}: Error finding container bb7bd871465d856415dd0b976be6d0f13aac5e1d4f726640e0788fab982cae20: Status 404 returned error can't find the container with id bb7bd871465d856415dd0b976be6d0f13aac5e1d4f726640e0788fab982cae20 Nov 03 22:26:07.152876 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.152840 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2f0612fd-99a8-4b86-8345-69dc450d4b75-archives-path\") pod \"periodic-gathering-95nwr-7rhld\" (UID: \"2f0612fd-99a8-4b86-8345-69dc450d4b75\") " pod="openshift-insights/periodic-gathering-95nwr-7rhld" Nov 03 22:26:07.153071 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.152887 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f0612fd-99a8-4b86-8345-69dc450d4b75-service-ca-bundle\") pod \"periodic-gathering-95nwr-7rhld\" (UID: \"2f0612fd-99a8-4b86-8345-69dc450d4b75\") " pod="openshift-insights/periodic-gathering-95nwr-7rhld" Nov 03 22:26:07.153071 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.152911 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f0612fd-99a8-4b86-8345-69dc450d4b75-trusted-ca-bundle\") pod \"periodic-gathering-95nwr-7rhld\" (UID: \"2f0612fd-99a8-4b86-8345-69dc450d4b75\") " pod="openshift-insights/periodic-gathering-95nwr-7rhld" Nov 03 22:26:07.153071 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.152993 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxzxd\" (UniqueName: \"kubernetes.io/projected/2f0612fd-99a8-4b86-8345-69dc450d4b75-kube-api-access-bxzxd\") pod \"periodic-gathering-95nwr-7rhld\" (UID: \"2f0612fd-99a8-4b86-8345-69dc450d4b75\") " pod="openshift-insights/periodic-gathering-95nwr-7rhld" Nov 03 22:26:07.153599 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.153569 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2f0612fd-99a8-4b86-8345-69dc450d4b75-archives-path\") pod \"periodic-gathering-95nwr-7rhld\" (UID: \"2f0612fd-99a8-4b86-8345-69dc450d4b75\") " pod="openshift-insights/periodic-gathering-95nwr-7rhld" Nov 03 22:26:07.153857 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.153831 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f0612fd-99a8-4b86-8345-69dc450d4b75-service-ca-bundle\") pod \"periodic-gathering-95nwr-7rhld\" (UID: \"2f0612fd-99a8-4b86-8345-69dc450d4b75\") " pod="openshift-insights/periodic-gathering-95nwr-7rhld" Nov 03 22:26:07.154041 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.154020 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f0612fd-99a8-4b86-8345-69dc450d4b75-trusted-ca-bundle\") pod \"periodic-gathering-95nwr-7rhld\" (UID: \"2f0612fd-99a8-4b86-8345-69dc450d4b75\") " pod="openshift-insights/periodic-gathering-95nwr-7rhld" Nov 03 22:26:07.213848 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.213753 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxzxd\" (UniqueName: \"kubernetes.io/projected/2f0612fd-99a8-4b86-8345-69dc450d4b75-kube-api-access-bxzxd\") pod \"periodic-gathering-95nwr-7rhld\" (UID: \"2f0612fd-99a8-4b86-8345-69dc450d4b75\") " pod="openshift-insights/periodic-gathering-95nwr-7rhld" Nov 03 22:26:07.253353 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.253304 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-95nwr-7rhld" Nov 03 22:26:07.638168 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.638128 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-95nwr-7rhld"] Nov 03 22:26:07.639720 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:26:07.639688 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0612fd_99a8_4b86_8345_69dc450d4b75.slice/crio-6fb7ea69457443104734a83c21b1664fecae1c2c3d1d04240d243089a7ad71e9 WatchSource:0}: Error finding container 6fb7ea69457443104734a83c21b1664fecae1c2c3d1d04240d243089a7ad71e9: Status 404 returned error can't find the container with id 6fb7ea69457443104734a83c21b1664fecae1c2c3d1d04240d243089a7ad71e9 Nov 03 22:26:07.873814 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.873751 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" event={"ID":"3b3927af-60a2-4432-99d6-b999a96f315f","Type":"ContainerStarted","Data":"1c02700858b888b781d85e71eae50a2b68567fc2b9d0cc2837fc06b4c32e3dd8"} Nov 03 22:26:07.873814 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.873817 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" event={"ID":"3b3927af-60a2-4432-99d6-b999a96f315f","Type":"ContainerStarted","Data":"bb7bd871465d856415dd0b976be6d0f13aac5e1d4f726640e0788fab982cae20"} Nov 03 22:26:07.875472 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.875443 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-95nwr-7rhld" event={"ID":"2f0612fd-99a8-4b86-8345-69dc450d4b75","Type":"ContainerStarted","Data":"31df60232f60bf445c11236f9fdec9a61bf66652d992691d1971b6e3d712e749"} Nov 03 22:26:07.875631 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.875501 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-95nwr-7rhld" event={"ID":"2f0612fd-99a8-4b86-8345-69dc450d4b75","Type":"ContainerStarted","Data":"6fb7ea69457443104734a83c21b1664fecae1c2c3d1d04240d243089a7ad71e9"} Nov 03 22:26:07.993938 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:07.992916 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" podStartSLOduration=1.992896161 podStartE2EDuration="1.992896161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:07.989716272 +0000 UTC m=+1135.583748246" watchObservedRunningTime="2026-11-03 22:26:07.992896161 +0000 UTC m=+1135.586928135" Nov 03 22:26:08.110094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:08.110010 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-95nwr-7rhld" podStartSLOduration=2.109916216 podStartE2EDuration="2.109916216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:08.108297061 +0000 UTC m=+1135.702329038" watchObservedRunningTime="2026-11-03 22:26:08.109916216 +0000 UTC m=+1135.703948193" Nov 03 22:26:08.651185 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:08.651143 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:12.060525 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:26:12.060485 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:26:16.650915 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:16.650869 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:16.664479 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:16.664446 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:16.950777 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:16.950691 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:26:35.901376 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:35.901340 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb"] Nov 03 22:26:35.926156 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:35.922260 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:35.975620 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:35.975585 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-906673-openshift-default-dockercfg-cwtcb\"" Nov 03 22:26:36.005801 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.005750 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca0b679-54b7-44fe-8a91-94de18621666-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.006007 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.005936 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca0b679-54b7-44fe-8a91-94de18621666-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.006120 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.006026 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.006120 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.006104 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbgt5\" (UniqueName: \"kubernetes.io/projected/5ca0b679-54b7-44fe-8a91-94de18621666-kube-api-access-lbgt5\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.006246 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.006155 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.006246 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.006224 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.006351 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.006254 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca0b679-54b7-44fe-8a91-94de18621666-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.006351 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.006282 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.006351 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.006307 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.058897 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.058853 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb"] Nov 03 22:26:36.110117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.108778 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.110117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.108864 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbgt5\" (UniqueName: \"kubernetes.io/projected/5ca0b679-54b7-44fe-8a91-94de18621666-kube-api-access-lbgt5\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.110117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.108921 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.110117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.108988 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.110117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.109023 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca0b679-54b7-44fe-8a91-94de18621666-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.110117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.109054 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.110117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.109126 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.110117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.109160 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca0b679-54b7-44fe-8a91-94de18621666-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.110117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.109266 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca0b679-54b7-44fe-8a91-94de18621666-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.133104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.118560 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.133104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.118851 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.133104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.128942 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca0b679-54b7-44fe-8a91-94de18621666-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.133104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.129243 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.133104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.129492 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.139591 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.139557 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.139942 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.139918 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca0b679-54b7-44fe-8a91-94de18621666-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.176765 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.176690 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbgt5\" (UniqueName: \"kubernetes.io/projected/5ca0b679-54b7-44fe-8a91-94de18621666-kube-api-access-lbgt5\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.187533 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.187480 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca0b679-54b7-44fe-8a91-94de18621666-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.262924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.262647 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:36.805798 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.805738 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb"] Nov 03 22:26:36.836696 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:26:36.836658 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca0b679_54b7_44fe_8a91_94de18621666.slice/crio-ed21ab1bdf35f1057e6212fca359d39e8e17d105631d0ebdcee407b2cb30df7f WatchSource:0}: Error finding container ed21ab1bdf35f1057e6212fca359d39e8e17d105631d0ebdcee407b2cb30df7f: Status 404 returned error can't find the container with id ed21ab1bdf35f1057e6212fca359d39e8e17d105631d0ebdcee407b2cb30df7f Nov 03 22:26:36.839464 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.839412 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:26:36.839590 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.839524 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:26:36.839590 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:36.839571 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:26:37.096144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:37.086559 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" event={"ID":"5ca0b679-54b7-44fe-8a91-94de18621666","Type":"ContainerStarted","Data":"0d9b92a8c9a0d02f96acb1a5695a01f684ebc850f6359f29760f9fab3e1e70c3"} Nov 03 22:26:37.096144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:37.086636 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" event={"ID":"5ca0b679-54b7-44fe-8a91-94de18621666","Type":"ContainerStarted","Data":"ed21ab1bdf35f1057e6212fca359d39e8e17d105631d0ebdcee407b2cb30df7f"} Nov 03 22:26:37.205562 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:37.205507 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" podStartSLOduration=2.205492626 podStartE2EDuration="2.205492626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:37.204757664 +0000 UTC m=+1164.798789650" watchObservedRunningTime="2026-11-03 22:26:37.205492626 +0000 UTC m=+1164.799524601" Nov 03 22:26:37.263540 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:37.263503 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:37.266250 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:37.266214 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:26:37.266506 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:37.266476 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" podUID="5ca0b679-54b7-44fe-8a91-94de18621666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 22:26:38.273153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:38.266623 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:26:38.273153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:38.266686 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" podUID="5ca0b679-54b7-44fe-8a91-94de18621666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 22:26:39.265609 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:39.265566 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:26:39.266367 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:39.265927 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" podUID="5ca0b679-54b7-44fe-8a91-94de18621666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 22:26:40.267719 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:40.267673 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:26:40.268393 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:40.268354 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" podUID="5ca0b679-54b7-44fe-8a91-94de18621666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 22:26:41.264102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:41.263590 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:26:41.264102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:41.263657 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" podUID="5ca0b679-54b7-44fe-8a91-94de18621666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 22:26:42.024479 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:26:42.024442 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:26:42.270748 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:42.270696 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:26:42.271092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:42.271023 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" podUID="5ca0b679-54b7-44fe-8a91-94de18621666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 22:26:43.263381 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:43.263335 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 22:26:43.263852 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:43.263415 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" podUID="5ca0b679-54b7-44fe-8a91-94de18621666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 22:26:44.268673 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:44.268638 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:44.269203 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:44.269166 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:26:44.269944 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:26:44.269920 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:27:12.026998 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:27:12.026949 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:27:12.783912 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:12.783882 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:27:12.787141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:12.787113 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:27:32.723796 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.723707 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7"] Nov 03 22:27:32.727914 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.727886 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:32.786049 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.786010 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7"] Nov 03 22:27:32.829802 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.829756 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"kube-root-ca.crt\"" Nov 03 22:27:32.833262 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.833234 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"openshift-service-ca.crt\"" Nov 03 22:27:32.833262 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.833234 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"distinct-2-w3-41bb2b-kserve-self-signed-certs\"" Nov 03 22:27:32.833262 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.833233 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-41bb2b\"/\"default-dockercfg-kdmdn\"" Nov 03 22:27:32.858396 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.858350 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-tmp-dir\") pod \"distinct-2-w3-41bb2b-kserve-846694fc58-2smc7\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:32.858584 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.858405 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-home\") pod \"distinct-2-w3-41bb2b-kserve-846694fc58-2smc7\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:32.858626 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.858575 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-model-cache\") pod \"distinct-2-w3-41bb2b-kserve-846694fc58-2smc7\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:32.858724 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.858703 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-tls-certs\") pod \"distinct-2-w3-41bb2b-kserve-846694fc58-2smc7\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:32.858778 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.858763 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc6w6\" (UniqueName: \"kubernetes.io/projected/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-kube-api-access-kc6w6\") pod \"distinct-2-w3-41bb2b-kserve-846694fc58-2smc7\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:32.858826 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.858788 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-dshm\") pod \"distinct-2-w3-41bb2b-kserve-846694fc58-2smc7\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:32.959931 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.959895 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-model-cache\") pod \"distinct-2-w3-41bb2b-kserve-846694fc58-2smc7\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:32.960155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.959969 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-tls-certs\") pod \"distinct-2-w3-41bb2b-kserve-846694fc58-2smc7\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:32.960155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.960011 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc6w6\" (UniqueName: \"kubernetes.io/projected/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-kube-api-access-kc6w6\") pod \"distinct-2-w3-41bb2b-kserve-846694fc58-2smc7\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:32.960155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.960032 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-dshm\") pod \"distinct-2-w3-41bb2b-kserve-846694fc58-2smc7\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:32.960155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.960147 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-tmp-dir\") pod \"distinct-2-w3-41bb2b-kserve-846694fc58-2smc7\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:32.960385 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.960177 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-home\") pod \"distinct-2-w3-41bb2b-kserve-846694fc58-2smc7\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:32.960561 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.960518 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-home\") pod \"distinct-2-w3-41bb2b-kserve-846694fc58-2smc7\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:32.960561 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.960535 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-model-cache\") pod \"distinct-2-w3-41bb2b-kserve-846694fc58-2smc7\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:32.960755 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.960604 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-tmp-dir\") pod \"distinct-2-w3-41bb2b-kserve-846694fc58-2smc7\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:32.962696 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.962666 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-tls-certs\") pod \"distinct-2-w3-41bb2b-kserve-846694fc58-2smc7\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:32.962832 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:32.962775 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-dshm\") pod \"distinct-2-w3-41bb2b-kserve-846694fc58-2smc7\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:33.017870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:33.017825 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc6w6\" (UniqueName: \"kubernetes.io/projected/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-kube-api-access-kc6w6\") pod \"distinct-2-w3-41bb2b-kserve-846694fc58-2smc7\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:33.040617 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:33.040578 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:33.499849 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:33.499820 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7"] Nov 03 22:27:33.501134 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:27:33.501047 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b0fcd09_e958_4cbb_9881_c6e6488cb39d.slice/crio-9f779940a871687b82dc5d2e7fc5b7038213f48dfceaa967f33463ab3a0f1c59 WatchSource:0}: Error finding container 9f779940a871687b82dc5d2e7fc5b7038213f48dfceaa967f33463ab3a0f1c59: Status 404 returned error can't find the container with id 9f779940a871687b82dc5d2e7fc5b7038213f48dfceaa967f33463ab3a0f1c59 Nov 03 22:27:33.536677 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:33.536639 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" event={"ID":"8b0fcd09-e958-4cbb-9881-c6e6488cb39d","Type":"ContainerStarted","Data":"9f779940a871687b82dc5d2e7fc5b7038213f48dfceaa967f33463ab3a0f1c59"} Nov 03 22:27:34.545261 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:34.545225 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" event={"ID":"8b0fcd09-e958-4cbb-9881-c6e6488cb39d","Type":"ContainerStarted","Data":"29e4a30937d211502610c2ac9b6d52353c9c9fdd318c5c9ab672944c09770478"} Nov 03 22:27:34.661197 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:34.661141 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podStartSLOduration=2.661127254 podStartE2EDuration="2.661127254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:34.659679654 +0000 UTC m=+1222.253711629" watchObservedRunningTime="2026-11-03 22:27:34.661127254 +0000 UTC m=+1222.255159227" Nov 03 22:27:35.041722 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:35.041687 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:37.685529 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:37.685483 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2"] Nov 03 22:27:37.689602 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:37.689581 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2" Nov 03 22:27:37.713343 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:37.713299 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10378c78-d3be-461e-ad15-ff38ecf8d7ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79aa88d1-h6zw2\" (UID: \"10378c78-d3be-461e-ad15-ff38ecf8d7ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2" Nov 03 22:27:37.713545 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:37.713415 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8srj2\" (UniqueName: \"kubernetes.io/projected/10378c78-d3be-461e-ad15-ff38ecf8d7ea-kube-api-access-8srj2\") pod \"maas-api-revoke-sub-79aa88d1-h6zw2\" (UID: \"10378c78-d3be-461e-ad15-ff38ecf8d7ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2" Nov 03 22:27:37.713545 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:37.713492 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10378c78-d3be-461e-ad15-ff38ecf8d7ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79aa88d1-h6zw2\" (UID: \"10378c78-d3be-461e-ad15-ff38ecf8d7ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2" Nov 03 22:27:37.741001 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:37.740968 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2"] Nov 03 22:27:37.788767 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:37.788731 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:27:37.788767 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:37.788734 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:27:37.814565 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:37.814523 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10378c78-d3be-461e-ad15-ff38ecf8d7ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79aa88d1-h6zw2\" (UID: \"10378c78-d3be-461e-ad15-ff38ecf8d7ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2" Nov 03 22:27:37.814772 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:37.814579 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8srj2\" (UniqueName: \"kubernetes.io/projected/10378c78-d3be-461e-ad15-ff38ecf8d7ea-kube-api-access-8srj2\") pod \"maas-api-revoke-sub-79aa88d1-h6zw2\" (UID: \"10378c78-d3be-461e-ad15-ff38ecf8d7ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2" Nov 03 22:27:37.814772 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:37.814630 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10378c78-d3be-461e-ad15-ff38ecf8d7ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79aa88d1-h6zw2\" (UID: \"10378c78-d3be-461e-ad15-ff38ecf8d7ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2" Nov 03 22:27:37.815329 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:37.815299 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10378c78-d3be-461e-ad15-ff38ecf8d7ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79aa88d1-h6zw2\" (UID: \"10378c78-d3be-461e-ad15-ff38ecf8d7ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2" Nov 03 22:27:37.817007 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:37.816984 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10378c78-d3be-461e-ad15-ff38ecf8d7ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79aa88d1-h6zw2\" (UID: \"10378c78-d3be-461e-ad15-ff38ecf8d7ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2" Nov 03 22:27:37.872836 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:37.872794 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8srj2\" (UniqueName: \"kubernetes.io/projected/10378c78-d3be-461e-ad15-ff38ecf8d7ea-kube-api-access-8srj2\") pod \"maas-api-revoke-sub-79aa88d1-h6zw2\" (UID: \"10378c78-d3be-461e-ad15-ff38ecf8d7ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2" Nov 03 22:27:38.005664 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:38.005619 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2" Nov 03 22:27:38.388611 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:38.388575 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2"] Nov 03 22:27:38.389379 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:27:38.389353 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10378c78_d3be_461e_ad15_ff38ecf8d7ea.slice/crio-7cb393bf93e21da076b46e5c789f5b39f43742e7484b314000c142dd31ec8bb5 WatchSource:0}: Error finding container 7cb393bf93e21da076b46e5c789f5b39f43742e7484b314000c142dd31ec8bb5: Status 404 returned error can't find the container with id 7cb393bf93e21da076b46e5c789f5b39f43742e7484b314000c142dd31ec8bb5 Nov 03 22:27:38.573577 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:38.573544 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2" event={"ID":"10378c78-d3be-461e-ad15-ff38ecf8d7ea","Type":"ContainerStarted","Data":"7cb393bf93e21da076b46e5c789f5b39f43742e7484b314000c142dd31ec8bb5"} Nov 03 22:27:39.582507 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:39.582465 5084 generic.go:356] "Generic (PLEG): container finished" podID="10378c78-d3be-461e-ad15-ff38ecf8d7ea" containerID="45a77e9fe20b49bcad9801ba972560bcb684924f940d7a86b2ece6e65500b05f" exitCode=0 Nov 03 22:27:39.582920 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:39.582544 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2" event={"ID":"10378c78-d3be-461e-ad15-ff38ecf8d7ea","Type":"ContainerDied","Data":"45a77e9fe20b49bcad9801ba972560bcb684924f940d7a86b2ece6e65500b05f"} Nov 03 22:27:40.757764 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:40.757723 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2" Nov 03 22:27:40.764796 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:40.764755 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10378c78-d3be-461e-ad15-ff38ecf8d7ea-kube-api-access-8srj2\" (UniqueName: \"kubernetes.io/projected/10378c78-d3be-461e-ad15-ff38ecf8d7ea-kube-api-access-8srj2\") pod \"10378c78-d3be-461e-ad15-ff38ecf8d7ea\" (UID: \"10378c78-d3be-461e-ad15-ff38ecf8d7ea\") " Nov 03 22:27:40.765216 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:40.765163 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10378c78-d3be-461e-ad15-ff38ecf8d7ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10378c78-d3be-461e-ad15-ff38ecf8d7ea-maas-api-openshift-ca\") pod \"10378c78-d3be-461e-ad15-ff38ecf8d7ea\" (UID: \"10378c78-d3be-461e-ad15-ff38ecf8d7ea\") " Nov 03 22:27:40.765659 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:40.765616 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10378c78-d3be-461e-ad15-ff38ecf8d7ea-maas-api-openshift-ca" pod "10378c78-d3be-461e-ad15-ff38ecf8d7ea" (UID: "10378c78-d3be-461e-ad15-ff38ecf8d7ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:27:40.765869 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:40.765637 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10378c78-d3be-461e-ad15-ff38ecf8d7ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10378c78-d3be-461e-ad15-ff38ecf8d7ea-maas-api-xks-ca\") pod \"10378c78-d3be-461e-ad15-ff38ecf8d7ea\" (UID: \"10378c78-d3be-461e-ad15-ff38ecf8d7ea\") " Nov 03 22:27:40.766594 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:40.766566 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10378c78-d3be-461e-ad15-ff38ecf8d7ea-maas-api-openshift-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:27:40.768257 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:40.768232 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10378c78-d3be-461e-ad15-ff38ecf8d7ea-kube-api-access-8srj2" pod "10378c78-d3be-461e-ad15-ff38ecf8d7ea" (UID: "10378c78-d3be-461e-ad15-ff38ecf8d7ea"). InnerVolumeSpecName "kube-api-access-8srj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:40.769960 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:40.769934 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10378c78-d3be-461e-ad15-ff38ecf8d7ea-maas-api-xks-ca" pod "10378c78-d3be-461e-ad15-ff38ecf8d7ea" (UID: "10378c78-d3be-461e-ad15-ff38ecf8d7ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:27:40.868445 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:40.868337 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10378c78-d3be-461e-ad15-ff38ecf8d7ea-maas-api-xks-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:27:40.868445 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:40.868382 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8srj2\" (UniqueName: \"kubernetes.io/projected/10378c78-d3be-461e-ad15-ff38ecf8d7ea-kube-api-access-8srj2\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:27:41.598208 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:41.598170 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2" event={"ID":"10378c78-d3be-461e-ad15-ff38ecf8d7ea","Type":"ContainerDied","Data":"7cb393bf93e21da076b46e5c789f5b39f43742e7484b314000c142dd31ec8bb5"} Nov 03 22:27:41.598208 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:41.598203 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2" Nov 03 22:27:41.598437 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:41.598221 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cb393bf93e21da076b46e5c789f5b39f43742e7484b314000c142dd31ec8bb5" Nov 03 22:27:42.028608 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:27:42.028575 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:27:43.041769 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:43.041737 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:43.054890 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:43.054851 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:43.614285 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:43.614244 5084 generic.go:356] "Generic (PLEG): container finished" podID="2f0612fd-99a8-4b86-8345-69dc450d4b75" containerID="31df60232f60bf445c11236f9fdec9a61bf66652d992691d1971b6e3d712e749" exitCode=0 Nov 03 22:27:43.614517 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:43.614311 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-95nwr-7rhld" event={"ID":"2f0612fd-99a8-4b86-8345-69dc450d4b75","Type":"ContainerDied","Data":"31df60232f60bf445c11236f9fdec9a61bf66652d992691d1971b6e3d712e749"} Nov 03 22:27:43.630893 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:43.630862 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:27:44.830617 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:44.830588 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-95nwr-7rhld" Nov 03 22:27:44.917740 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:44.917648 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f0612fd-99a8-4b86-8345-69dc450d4b75-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f0612fd-99a8-4b86-8345-69dc450d4b75-trusted-ca-bundle\") pod \"2f0612fd-99a8-4b86-8345-69dc450d4b75\" (UID: \"2f0612fd-99a8-4b86-8345-69dc450d4b75\") " Nov 03 22:27:44.917740 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:44.917693 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f0612fd-99a8-4b86-8345-69dc450d4b75-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f0612fd-99a8-4b86-8345-69dc450d4b75-service-ca-bundle\") pod \"2f0612fd-99a8-4b86-8345-69dc450d4b75\" (UID: \"2f0612fd-99a8-4b86-8345-69dc450d4b75\") " Nov 03 22:27:44.917995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:44.917785 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0612fd-99a8-4b86-8345-69dc450d4b75-kube-api-access-bxzxd\" (UniqueName: \"kubernetes.io/projected/2f0612fd-99a8-4b86-8345-69dc450d4b75-kube-api-access-bxzxd\") pod \"2f0612fd-99a8-4b86-8345-69dc450d4b75\" (UID: \"2f0612fd-99a8-4b86-8345-69dc450d4b75\") " Nov 03 22:27:44.917995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:44.917834 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0612fd-99a8-4b86-8345-69dc450d4b75-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2f0612fd-99a8-4b86-8345-69dc450d4b75-archives-path\") pod \"2f0612fd-99a8-4b86-8345-69dc450d4b75\" (UID: \"2f0612fd-99a8-4b86-8345-69dc450d4b75\") " Nov 03 22:27:44.918159 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:44.918096 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0612fd-99a8-4b86-8345-69dc450d4b75-service-ca-bundle" pod "2f0612fd-99a8-4b86-8345-69dc450d4b75" (UID: "2f0612fd-99a8-4b86-8345-69dc450d4b75"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:27:44.918159 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:44.918143 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0612fd-99a8-4b86-8345-69dc450d4b75-archives-path" pod "2f0612fd-99a8-4b86-8345-69dc450d4b75" (UID: "2f0612fd-99a8-4b86-8345-69dc450d4b75"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:27:44.918262 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:44.918155 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0612fd-99a8-4b86-8345-69dc450d4b75-trusted-ca-bundle" pod "2f0612fd-99a8-4b86-8345-69dc450d4b75" (UID: "2f0612fd-99a8-4b86-8345-69dc450d4b75"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:27:44.918317 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:44.918272 5084 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2f0612fd-99a8-4b86-8345-69dc450d4b75-archives-path\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:27:44.918317 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:44.918293 5084 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f0612fd-99a8-4b86-8345-69dc450d4b75-trusted-ca-bundle\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:27:44.918317 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:44.918308 5084 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f0612fd-99a8-4b86-8345-69dc450d4b75-service-ca-bundle\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:27:44.920110 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:44.920032 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0612fd-99a8-4b86-8345-69dc450d4b75-kube-api-access-bxzxd" pod "2f0612fd-99a8-4b86-8345-69dc450d4b75" (UID: "2f0612fd-99a8-4b86-8345-69dc450d4b75"). InnerVolumeSpecName "kube-api-access-bxzxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:45.019612 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:45.019571 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxzxd\" (UniqueName: \"kubernetes.io/projected/2f0612fd-99a8-4b86-8345-69dc450d4b75-kube-api-access-bxzxd\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:27:45.630398 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:45.630364 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-95nwr-7rhld" Nov 03 22:27:45.630398 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:45.630378 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-95nwr-7rhld" event={"ID":"2f0612fd-99a8-4b86-8345-69dc450d4b75","Type":"ContainerDied","Data":"6fb7ea69457443104734a83c21b1664fecae1c2c3d1d04240d243089a7ad71e9"} Nov 03 22:27:45.630638 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:45.630434 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fb7ea69457443104734a83c21b1664fecae1c2c3d1d04240d243089a7ad71e9" Nov 03 22:27:50.646951 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.646903 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz"] Nov 03 22:27:50.647464 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.647447 5084 cpu_manager.go:414] "containerMap: removing container" podUID="10378c78-d3be-461e-ad15-ff38ecf8d7ea" containerName="revoke-keys" Nov 03 22:27:50.647518 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.647465 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10378c78-d3be-461e-ad15-ff38ecf8d7ea" containerName="revoke-keys" Nov 03 22:27:50.647518 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.647508 5084 cpu_manager.go:414] "containerMap: removing container" podUID="2f0612fd-99a8-4b86-8345-69dc450d4b75" containerName="insights-gathering" Nov 03 22:27:50.647518 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.647513 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0612fd-99a8-4b86-8345-69dc450d4b75" containerName="insights-gathering" Nov 03 22:27:50.647609 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.647600 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0612fd-99a8-4b86-8345-69dc450d4b75" containerName="insights-gathering" Nov 03 22:27:50.647650 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.647609 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="10378c78-d3be-461e-ad15-ff38ecf8d7ea" containerName="revoke-keys" Nov 03 22:27:50.651435 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.651406 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.711244 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.711196 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.711441 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.711249 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.711441 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.711403 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.711592 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.711566 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.711687 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.711665 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.711738 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.711723 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.711797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.711778 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.711952 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.711932 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f6t5\" (UniqueName: \"kubernetes.io/projected/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-kube-api-access-4f6t5\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.712101 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.711989 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.762368 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.762318 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz"] Nov 03 22:27:50.813105 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.813019 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.813105 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.813091 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.813376 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.813154 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.813376 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.813188 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.813376 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.813226 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.813376 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.813263 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.813376 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.813334 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f6t5\" (UniqueName: \"kubernetes.io/projected/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-kube-api-access-4f6t5\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.813376 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.813369 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.813694 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.813464 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.813694 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.813500 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.813694 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.813529 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.813694 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.813582 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.813892 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.813787 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.814026 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.814005 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.815635 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.815616 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.815793 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.815776 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.871414 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.871380 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.871683 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.871656 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f6t5\" (UniqueName: \"kubernetes.io/projected/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-kube-api-access-4f6t5\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:50.966589 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:50.966486 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:51.411419 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:51.411390 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz"] Nov 03 22:27:51.411989 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:27:51.411967 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ba9a52_6593_4b44_8b5a_5ed4f1fd0b61.slice/crio-ba6f036e236c586f656cc47ca757617bff45beed711c971acbca903ba7317637 WatchSource:0}: Error finding container ba6f036e236c586f656cc47ca757617bff45beed711c971acbca903ba7317637: Status 404 returned error can't find the container with id ba6f036e236c586f656cc47ca757617bff45beed711c971acbca903ba7317637 Nov 03 22:27:51.414413 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:51.414378 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:27:51.414506 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:51.414451 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:27:51.414506 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:51.414481 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:27:51.738939 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:51.738827 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" event={"ID":"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61","Type":"ContainerStarted","Data":"b621cc8e9708fac69fc6f048357c0de7a270915b2207fa3bfa3eefc764b0491b"} Nov 03 22:27:51.738939 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:51.738901 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" event={"ID":"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61","Type":"ContainerStarted","Data":"ba6f036e236c586f656cc47ca757617bff45beed711c971acbca903ba7317637"} Nov 03 22:27:51.862320 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:51.862258 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" podStartSLOduration=1.862240487 podStartE2EDuration="1.862240487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:51.858803689 +0000 UTC m=+1239.452835665" watchObservedRunningTime="2026-11-03 22:27:51.862240487 +0000 UTC m=+1239.456272461" Nov 03 22:27:51.967076 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:51.967023 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:52.970087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:52.969979 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:27:52.970621 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:52.970091 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" podUID="b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 22:27:53.897564 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:53.897516 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj"] Nov 03 22:27:53.908526 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:53.908413 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:53.923101 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:53.923044 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj"] Nov 03 22:27:53.988370 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:53.988327 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f7e6db\"/\"distinct-2-w4-f7e6db-kserve-self-signed-certs\"" Nov 03 22:27:53.992204 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:53.988045 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:27:53.992428 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:53.992260 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" podUID="b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 22:27:54.092912 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.092864 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-tmp-dir\") pod \"distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.093283 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.093258 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl8nl\" (UniqueName: \"kubernetes.io/projected/8de619f3-ea48-44dc-b8a6-d70502f99760-kube-api-access-fl8nl\") pod \"distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.093530 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.093510 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-home\") pod \"distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.093787 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.093768 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de619f3-ea48-44dc-b8a6-d70502f99760-tls-certs\") pod \"distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.094010 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.093989 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-model-cache\") pod \"distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.094231 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.094213 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-dshm\") pod \"distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.195745 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.195635 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de619f3-ea48-44dc-b8a6-d70502f99760-tls-certs\") pod \"distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.196045 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.196024 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-model-cache\") pod \"distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.196235 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.196218 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-dshm\") pod \"distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.196661 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.196558 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-model-cache\") pod \"distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.197011 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.196991 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-tmp-dir\") pod \"distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.197646 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.197613 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl8nl\" (UniqueName: \"kubernetes.io/projected/8de619f3-ea48-44dc-b8a6-d70502f99760-kube-api-access-fl8nl\") pod \"distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.197749 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.197696 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-home\") pod \"distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.197857 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.197426 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-tmp-dir\") pod \"distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.198186 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.198162 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-home\") pod \"distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.199075 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.199033 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-dshm\") pod \"distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.199458 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.199433 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de619f3-ea48-44dc-b8a6-d70502f99760-tls-certs\") pod \"distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.258286 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.258214 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl8nl\" (UniqueName: \"kubernetes.io/projected/8de619f3-ea48-44dc-b8a6-d70502f99760-kube-api-access-fl8nl\") pod \"distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.533312 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.533269 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:54.953107 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.953011 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj"] Nov 03 22:27:54.954086 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:27:54.954029 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8de619f3_ea48_44dc_b8a6_d70502f99760.slice/crio-a7fa086bb7f2b8a9a02ce1f31d98cf069c3fb82c65a2d1f67ce823e1686a4b00 WatchSource:0}: Error finding container a7fa086bb7f2b8a9a02ce1f31d98cf069c3fb82c65a2d1f67ce823e1686a4b00: Status 404 returned error can't find the container with id a7fa086bb7f2b8a9a02ce1f31d98cf069c3fb82c65a2d1f67ce823e1686a4b00 Nov 03 22:27:54.967211 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.967176 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:27:54.967528 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:54.967441 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" podUID="b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 22:27:55.799220 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:55.799173 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" event={"ID":"8de619f3-ea48-44dc-b8a6-d70502f99760","Type":"ContainerStarted","Data":"058da7f6b7bb7eb57aee663e4737ef42f0190373218377565b86077d2f779d09"} Nov 03 22:27:55.799220 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:55.799220 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" event={"ID":"8de619f3-ea48-44dc-b8a6-d70502f99760","Type":"ContainerStarted","Data":"a7fa086bb7f2b8a9a02ce1f31d98cf069c3fb82c65a2d1f67ce823e1686a4b00"} Nov 03 22:27:55.967715 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:55.967591 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:27:55.967715 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:55.967659 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" podUID="b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 22:27:56.534338 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:56.534299 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:27:56.973153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:56.971567 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:27:56.973153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:56.971639 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" podUID="b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 22:27:57.967418 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:57.967363 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 22:27:57.967605 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:57.967447 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" podUID="b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 22:27:59.226853 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:59.225473 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:59.227429 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:59.227009 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:59.228025 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:59.228001 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:27:59.400850 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:27:59.400792 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" podStartSLOduration=6.400776684 podStartE2EDuration="6.400776684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:55.916589286 +0000 UTC m=+1243.510621260" watchObservedRunningTime="2026-11-03 22:27:59.400776684 +0000 UTC m=+1246.994808657" Nov 03 22:28:04.533992 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:04.533952 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:28:04.547236 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:04.547205 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:28:04.880303 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:04.880208 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:28:12.025842 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:28:12.025792 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:28:13.050449 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:13.050401 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2"] Nov 03 22:28:13.102772 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:13.102722 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88770b16-bcfd2"] Nov 03 22:28:14.653382 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:14.653341 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d" path="/var/lib/kubelet/pods/2c98c7dd-fdfc-4dc7-a7cc-c81e30cd376d/volumes" Nov 03 22:28:22.131716 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.131671 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2"] Nov 03 22:28:22.137115 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.137052 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.251485 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.251443 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2"] Nov 03 22:28:22.289389 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.289345 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdvh8\" (UniqueName: \"kubernetes.io/projected/5be671e7-45d4-4e28-b384-2f3dd6397ecb-kube-api-access-hdvh8\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.289579 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.289527 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.289625 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.289574 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.289660 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.289637 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.289709 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.289692 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.289775 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.289754 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.289822 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.289812 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.289858 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.289833 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.289893 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.289874 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.391586 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.391481 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.391586 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.391537 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.391586 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.391564 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.391905 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.391603 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdvh8\" (UniqueName: \"kubernetes.io/projected/5be671e7-45d4-4e28-b384-2f3dd6397ecb-kube-api-access-hdvh8\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.391905 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.391697 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.391905 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.391733 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.391905 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.391794 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.391905 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.391819 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.391905 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.391845 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.392307 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.392161 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-workload-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.392307 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.392282 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-data\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.392426 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.392339 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-credential-socket\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.392426 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.392361 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-workload-certs\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.392778 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.392754 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istiod-ca-cert\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.394103 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.394079 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-envoy\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.394258 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.394240 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-podinfo\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.449691 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.449653 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-token\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.449854 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.449828 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdvh8\" (UniqueName: \"kubernetes.io/projected/5be671e7-45d4-4e28-b384-2f3dd6397ecb-kube-api-access-hdvh8\") pod \"e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.452470 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.452434 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:22.750355 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.750262 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9"] Nov 03 22:28:22.755668 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.755637 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.811098 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.811033 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.811435 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.811408 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.811655 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.811631 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.811840 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.811822 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjj94\" (UniqueName: \"kubernetes.io/projected/531b6497-408c-4315-806e-0504eead114a-kube-api-access-gjj94\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.811968 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.811954 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/531b6497-408c-4315-806e-0504eead114a-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.812109 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.812094 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/531b6497-408c-4315-806e-0504eead114a-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.812264 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.812226 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.813509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.812309 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/531b6497-408c-4315-806e-0504eead114a-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.813509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.812357 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.818220 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.818182 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9"] Nov 03 22:28:22.914035 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.913989 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjj94\" (UniqueName: \"kubernetes.io/projected/531b6497-408c-4315-806e-0504eead114a-kube-api-access-gjj94\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.914162 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.914044 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/531b6497-408c-4315-806e-0504eead114a-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.914162 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.914103 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/531b6497-408c-4315-806e-0504eead114a-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.914162 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.914141 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.914433 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.914204 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/531b6497-408c-4315-806e-0504eead114a-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.914433 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.914391 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.914554 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.914494 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.914612 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.914600 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.914678 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.914631 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.914736 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.914696 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.914884 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.914858 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.915031 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.915011 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.915110 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.915011 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/531b6497-408c-4315-806e-0504eead114a-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.915274 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.915259 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.916109 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.916083 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2"] Nov 03 22:28:22.916962 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.916938 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/531b6497-408c-4315-806e-0504eead114a-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.917621 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.917597 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.920244 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.920208 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:28:22.920337 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.920284 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:28:22.920337 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.920315 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:28:22.973362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.973325 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/531b6497-408c-4315-806e-0504eead114a-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:22.973362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:22.973355 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjj94\" (UniqueName: \"kubernetes.io/projected/531b6497-408c-4315-806e-0504eead114a-kube-api-access-gjj94\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:23.008842 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.008728 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" event={"ID":"5be671e7-45d4-4e28-b384-2f3dd6397ecb","Type":"ContainerStarted","Data":"69842cadac3a1434e1507d203187c9f4d9e1c3658fae8953789d6d276467e1bd"} Nov 03 22:28:23.008842 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.008801 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" event={"ID":"5be671e7-45d4-4e28-b384-2f3dd6397ecb","Type":"ContainerStarted","Data":"32c9cf2e07671b6a409059112d462b415e5371a46b67b6263ab85bc2ad37d7a1"} Nov 03 22:28:23.071218 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.070884 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:23.132506 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.132432 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" podStartSLOduration=1.132410479 podStartE2EDuration="1.132410479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:23.126161332 +0000 UTC m=+1270.720193306" watchObservedRunningTime="2026-11-03 22:28:23.132410479 +0000 UTC m=+1270.726442459" Nov 03 22:28:23.231341 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.231305 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n"] Nov 03 22:28:23.237435 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.237403 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n" Nov 03 22:28:23.288198 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.288153 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n"] Nov 03 22:28:23.293334 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.291141 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:28:23.293334 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.291383 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:28:23.322979 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.322923 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12a34bd6-bj45n\" (UID: \"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n" Nov 03 22:28:23.323262 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.322999 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12a34bd6-bj45n\" (UID: \"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n" Nov 03 22:28:23.323262 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.323203 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-774n2\" (UniqueName: \"kubernetes.io/projected/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-kube-api-access-774n2\") pod \"maas-api-revoke-sub-12a34bd6-bj45n\" (UID: \"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n" Nov 03 22:28:23.424160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.424119 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12a34bd6-bj45n\" (UID: \"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n" Nov 03 22:28:23.424379 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.424183 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12a34bd6-bj45n\" (UID: \"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n" Nov 03 22:28:23.424379 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.424325 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-774n2\" (UniqueName: \"kubernetes.io/projected/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-kube-api-access-774n2\") pod \"maas-api-revoke-sub-12a34bd6-bj45n\" (UID: \"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n" Nov 03 22:28:23.424837 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.424815 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12a34bd6-bj45n\" (UID: \"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n" Nov 03 22:28:23.426946 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.426914 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12a34bd6-bj45n\" (UID: \"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n" Nov 03 22:28:23.453632 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.453591 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:23.455556 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.455524 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:28:23.455732 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.455583 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" podUID="5be671e7-45d4-4e28-b384-2f3dd6397ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 22:28:23.470670 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.470640 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9"] Nov 03 22:28:23.473001 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:28:23.472955 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531b6497_408c_4315_806e_0504eead114a.slice/crio-ada9b60eddd3fbcd1383f7d22308232872296d3544ad5ccfb07c45399b904bc6 WatchSource:0}: Error finding container ada9b60eddd3fbcd1383f7d22308232872296d3544ad5ccfb07c45399b904bc6: Status 404 returned error can't find the container with id ada9b60eddd3fbcd1383f7d22308232872296d3544ad5ccfb07c45399b904bc6 Nov 03 22:28:23.475929 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.475885 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:28:23.476113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.475974 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:28:23.476113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.476096 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:28:23.482832 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.482722 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-774n2\" (UniqueName: \"kubernetes.io/projected/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-kube-api-access-774n2\") pod \"maas-api-revoke-sub-12a34bd6-bj45n\" (UID: \"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n" Nov 03 22:28:23.583038 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:23.577725 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n" Nov 03 22:28:24.000653 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:24.000621 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n"] Nov 03 22:28:24.034369 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:24.034327 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" event={"ID":"531b6497-408c-4315-806e-0504eead114a","Type":"ContainerStarted","Data":"43b1c2e494111edaf6ad2141c39562f40b92641dd0c17ebf3eee031ab3e6d162"} Nov 03 22:28:24.034525 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:24.034401 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" event={"ID":"531b6497-408c-4315-806e-0504eead114a","Type":"ContainerStarted","Data":"ada9b60eddd3fbcd1383f7d22308232872296d3544ad5ccfb07c45399b904bc6"} Nov 03 22:28:24.037947 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:24.037914 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n" event={"ID":"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2","Type":"ContainerStarted","Data":"71ed727062476209f00cf2a4c9b011f7230ab1093d533bb43afade681f4254da"} Nov 03 22:28:24.076329 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:24.076172 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:24.076329 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:24.076310 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:28:24.076483 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:24.076354 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" podUID="531b6497-408c-4315-806e-0504eead114a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 22:28:24.191755 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:24.191678 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" podStartSLOduration=2.191656872 podStartE2EDuration="2.191656872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:24.154155316 +0000 UTC m=+1271.748187281" watchObservedRunningTime="2026-11-03 22:28:24.191656872 +0000 UTC m=+1271.785688847" Nov 03 22:28:24.453497 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:24.453456 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:28:24.453682 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:24.453510 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" podUID="5be671e7-45d4-4e28-b384-2f3dd6397ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 22:28:25.046683 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:25.046633 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n" event={"ID":"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2","Type":"ContainerStarted","Data":"defd66f90b7291da843bc2c77f1e1db6267bef309cefeb4fe89817179c52ec7b"} Nov 03 22:28:25.074487 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:25.073649 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:28:25.074487 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:25.073722 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" podUID="531b6497-408c-4315-806e-0504eead114a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 22:28:25.179088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:25.178523 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n" podStartSLOduration=2.178503529 podStartE2EDuration="2.178503529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:25.177542782 +0000 UTC m=+1272.771574756" watchObservedRunningTime="2026-11-03 22:28:25.178503529 +0000 UTC m=+1272.772535504" Nov 03 22:28:25.453266 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:25.453171 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:28:25.453266 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:25.453236 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" podUID="5be671e7-45d4-4e28-b384-2f3dd6397ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 22:28:26.060118 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:26.059373 5084 generic.go:356] "Generic (PLEG): container finished" podID="caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2" containerID="defd66f90b7291da843bc2c77f1e1db6267bef309cefeb4fe89817179c52ec7b" exitCode=0 Nov 03 22:28:26.064211 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:26.060759 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n" event={"ID":"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2","Type":"ContainerDied","Data":"defd66f90b7291da843bc2c77f1e1db6267bef309cefeb4fe89817179c52ec7b"} Nov 03 22:28:26.074091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:26.072540 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:28:26.074091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:26.072608 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" podUID="531b6497-408c-4315-806e-0504eead114a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 22:28:26.453270 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:26.453183 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:28:26.453270 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:26.453243 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" podUID="5be671e7-45d4-4e28-b384-2f3dd6397ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 22:28:27.076644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:27.076605 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:28:27.076930 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:27.076903 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" podUID="531b6497-408c-4315-806e-0504eead114a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 22:28:27.446028 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:27.445988 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n" Nov 03 22:28:27.452990 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:27.452956 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:28:27.453104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:27.453018 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" podUID="5be671e7-45d4-4e28-b384-2f3dd6397ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 22:28:27.523866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:27.523819 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-kube-api-access-774n2\" (UniqueName: \"kubernetes.io/projected/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-kube-api-access-774n2\") pod \"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2\" (UID: \"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2\") " Nov 03 22:28:27.524429 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:27.523962 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-maas-api-openshift-ca\") pod \"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2\" (UID: \"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2\") " Nov 03 22:28:27.524429 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:27.524111 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-maas-api-xks-ca\") pod \"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2\" (UID: \"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2\") " Nov 03 22:28:27.524429 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:27.524299 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-maas-api-openshift-ca" pod "caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2" (UID: "caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:27.524585 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:27.524569 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-maas-api-openshift-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:27.526156 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:27.526131 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-kube-api-access-774n2" pod "caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2" (UID: "caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2"). InnerVolumeSpecName "kube-api-access-774n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:27.526390 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:27.526367 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-maas-api-xks-ca" pod "caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2" (UID: "caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:27.625678 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:27.625572 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-774n2\" (UniqueName: \"kubernetes.io/projected/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-kube-api-access-774n2\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:27.625678 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:27.625624 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2-maas-api-xks-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:28.074088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:28.073596 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:28:28.074088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:28.073665 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" podUID="531b6497-408c-4315-806e-0504eead114a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 22:28:28.118478 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:28.118445 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n" Nov 03 22:28:28.118678 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:28.118473 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n" event={"ID":"caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2","Type":"ContainerDied","Data":"71ed727062476209f00cf2a4c9b011f7230ab1093d533bb43afade681f4254da"} Nov 03 22:28:28.118678 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:28.118522 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71ed727062476209f00cf2a4c9b011f7230ab1093d533bb43afade681f4254da" Nov 03 22:28:28.452865 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:28.452769 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:28:28.452865 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:28.452828 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" podUID="5be671e7-45d4-4e28-b384-2f3dd6397ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 22:28:29.072560 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:29.072514 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:28:29.073179 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:29.073143 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" podUID="531b6497-408c-4315-806e-0504eead114a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 22:28:29.453225 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:29.453136 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 22:28:29.453225 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:29.453207 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" podUID="5be671e7-45d4-4e28-b384-2f3dd6397ecb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 22:28:30.071924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:30.071878 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 22:28:30.072155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:30.071965 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" podUID="531b6497-408c-4315-806e-0504eead114a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 22:28:30.724767 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:30.724728 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:30.725445 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:30.725136 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:30.727002 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:30.726978 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:28:31.323682 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:31.323646 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:31.324513 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:31.324481 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:31.326316 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:31.326290 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:28:34.050411 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:34.050372 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8"] Nov 03 22:28:34.103350 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:34.103305 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c74e75a1-j6bk8"] Nov 03 22:28:34.630726 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:34.630687 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15ee554-111c-4eff-bfa0-98c40634e126" path="/var/lib/kubelet/pods/a15ee554-111c-4eff-bfa0-98c40634e126/volumes" Nov 03 22:28:42.022866 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:28:42.022829 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:28:50.049039 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.048997 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v"] Nov 03 22:28:50.099817 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.099779 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806f55be-m949v"] Nov 03 22:28:50.629787 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.629743 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbfb1096-ae10-4197-b046-ae9a310706e8" path="/var/lib/kubelet/pods/fbfb1096-ae10-4197-b046-ae9a310706e8/volumes" Nov 03 22:28:50.680418 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.680383 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6"] Nov 03 22:28:50.680962 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.680946 5084 cpu_manager.go:414] "containerMap: removing container" podUID="caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2" containerName="revoke-keys" Nov 03 22:28:50.681033 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.680964 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2" containerName="revoke-keys" Nov 03 22:28:50.681122 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.681094 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2" containerName="revoke-keys" Nov 03 22:28:50.685736 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.685708 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6" Nov 03 22:28:50.737986 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.737952 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6"] Nov 03 22:28:50.747975 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.747936 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f84bfa22-2h2p6\" (UID: \"4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6" Nov 03 22:28:50.748277 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.748249 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f84bfa22-2h2p6\" (UID: \"4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6" Nov 03 22:28:50.748422 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.748406 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtz2j\" (UniqueName: \"kubernetes.io/projected/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-kube-api-access-mtz2j\") pod \"maas-api-revoke-sub-f84bfa22-2h2p6\" (UID: \"4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6" Nov 03 22:28:50.784682 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.784637 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:28:50.784900 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.784686 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:28:50.849681 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.849640 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtz2j\" (UniqueName: \"kubernetes.io/projected/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-kube-api-access-mtz2j\") pod \"maas-api-revoke-sub-f84bfa22-2h2p6\" (UID: \"4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6" Nov 03 22:28:50.849883 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.849776 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f84bfa22-2h2p6\" (UID: \"4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6" Nov 03 22:28:50.849883 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.849826 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f84bfa22-2h2p6\" (UID: \"4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6" Nov 03 22:28:50.850627 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.850594 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f84bfa22-2h2p6\" (UID: \"4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6" Nov 03 22:28:50.852546 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.852515 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f84bfa22-2h2p6\" (UID: \"4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6" Nov 03 22:28:50.908549 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:50.908467 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtz2j\" (UniqueName: \"kubernetes.io/projected/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-kube-api-access-mtz2j\") pod \"maas-api-revoke-sub-f84bfa22-2h2p6\" (UID: \"4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6" Nov 03 22:28:51.000432 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:51.000388 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6" Nov 03 22:28:51.384161 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:51.384131 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6"] Nov 03 22:28:51.385036 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:28:51.385008 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec16b4a_f5d0_4ced_ba33_3fc1ca1b9c34.slice/crio-7388685ab8d5254910b986e2c792e9a358cf4b564059d322aa8480514a4e4f9e WatchSource:0}: Error finding container 7388685ab8d5254910b986e2c792e9a358cf4b564059d322aa8480514a4e4f9e: Status 404 returned error can't find the container with id 7388685ab8d5254910b986e2c792e9a358cf4b564059d322aa8480514a4e4f9e Nov 03 22:28:52.395049 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:52.395013 5084 generic.go:356] "Generic (PLEG): container finished" podID="4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34" containerID="10a731127f79eb2a29d49ada679c91833d0671e8ac868b2415dd4502a394b443" exitCode=0 Nov 03 22:28:52.395631 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:52.395101 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6" event={"ID":"4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34","Type":"ContainerDied","Data":"10a731127f79eb2a29d49ada679c91833d0671e8ac868b2415dd4502a394b443"} Nov 03 22:28:52.395631 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:52.395170 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6" event={"ID":"4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34","Type":"ContainerStarted","Data":"7388685ab8d5254910b986e2c792e9a358cf4b564059d322aa8480514a4e4f9e"} Nov 03 22:28:53.572498 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:53.572465 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6" Nov 03 22:28:53.690682 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:53.690589 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-maas-api-openshift-ca\") pod \"4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34\" (UID: \"4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34\") " Nov 03 22:28:53.690977 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:53.690947 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-maas-api-xks-ca\") pod \"4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34\" (UID: \"4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34\") " Nov 03 22:28:53.691233 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:53.691143 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-kube-api-access-mtz2j\" (UniqueName: \"kubernetes.io/projected/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-kube-api-access-mtz2j\") pod \"4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34\" (UID: \"4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34\") " Nov 03 22:28:53.694146 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:53.694098 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-maas-api-openshift-ca" pod "4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34" (UID: "4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:53.715811 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:53.715769 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-kube-api-access-mtz2j" pod "4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34" (UID: "4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34"). InnerVolumeSpecName "kube-api-access-mtz2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:53.716496 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:53.716463 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-maas-api-xks-ca" pod "4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34" (UID: "4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:53.792726 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:53.792679 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-maas-api-openshift-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:53.792726 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:53.792730 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-maas-api-xks-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:53.792987 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:53.792747 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtz2j\" (UniqueName: \"kubernetes.io/projected/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34-kube-api-access-mtz2j\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:28:54.411739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:54.411700 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6" Nov 03 22:28:54.411739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:54.411733 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6" event={"ID":"4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34","Type":"ContainerDied","Data":"7388685ab8d5254910b986e2c792e9a358cf4b564059d322aa8480514a4e4f9e"} Nov 03 22:28:54.412084 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:28:54.411818 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7388685ab8d5254910b986e2c792e9a358cf4b564059d322aa8480514a4e4f9e" Nov 03 22:29:02.048775 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:02.048726 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5"] Nov 03 22:29:02.099793 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:02.099752 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae36204d-l59p5"] Nov 03 22:29:02.631029 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:02.630994 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37" path="/var/lib/kubelet/pods/ef4dfc89-2d3d-4fda-9651-7be3d5cfcb37/volumes" Nov 03 22:29:06.327944 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:06.327894 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh"] Nov 03 22:29:06.328818 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:06.328792 5084 cpu_manager.go:414] "containerMap: removing container" podUID="4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34" containerName="revoke-keys" Nov 03 22:29:06.328881 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:06.328822 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34" containerName="revoke-keys" Nov 03 22:29:06.329056 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:06.329040 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34" containerName="revoke-keys" Nov 03 22:29:06.333488 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:06.333453 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh" Nov 03 22:29:06.355188 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:06.355152 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d997c646-e8fd-4a08-99b6-603ab21587a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49a58bb0-t2hnh\" (UID: \"d997c646-e8fd-4a08-99b6-603ab21587a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh" Nov 03 22:29:06.355188 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:06.355187 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d997c646-e8fd-4a08-99b6-603ab21587a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49a58bb0-t2hnh\" (UID: \"d997c646-e8fd-4a08-99b6-603ab21587a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh" Nov 03 22:29:06.355452 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:06.355232 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gcqh\" (UniqueName: \"kubernetes.io/projected/d997c646-e8fd-4a08-99b6-603ab21587a7-kube-api-access-5gcqh\") pod \"maas-api-revoke-sub-49a58bb0-t2hnh\" (UID: \"d997c646-e8fd-4a08-99b6-603ab21587a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh" Nov 03 22:29:06.383356 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:06.383299 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh"] Nov 03 22:29:06.430510 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:06.430472 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:29:06.430510 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:06.430508 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:29:06.456214 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:06.456177 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d997c646-e8fd-4a08-99b6-603ab21587a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49a58bb0-t2hnh\" (UID: \"d997c646-e8fd-4a08-99b6-603ab21587a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh" Nov 03 22:29:06.456214 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:06.456215 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d997c646-e8fd-4a08-99b6-603ab21587a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49a58bb0-t2hnh\" (UID: \"d997c646-e8fd-4a08-99b6-603ab21587a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh" Nov 03 22:29:06.456478 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:06.456262 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gcqh\" (UniqueName: \"kubernetes.io/projected/d997c646-e8fd-4a08-99b6-603ab21587a7-kube-api-access-5gcqh\") pod \"maas-api-revoke-sub-49a58bb0-t2hnh\" (UID: \"d997c646-e8fd-4a08-99b6-603ab21587a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh" Nov 03 22:29:06.457053 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:06.457032 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d997c646-e8fd-4a08-99b6-603ab21587a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49a58bb0-t2hnh\" (UID: \"d997c646-e8fd-4a08-99b6-603ab21587a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh" Nov 03 22:29:06.459197 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:06.459164 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d997c646-e8fd-4a08-99b6-603ab21587a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49a58bb0-t2hnh\" (UID: \"d997c646-e8fd-4a08-99b6-603ab21587a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh" Nov 03 22:29:06.514851 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:06.514818 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gcqh\" (UniqueName: \"kubernetes.io/projected/d997c646-e8fd-4a08-99b6-603ab21587a7-kube-api-access-5gcqh\") pod \"maas-api-revoke-sub-49a58bb0-t2hnh\" (UID: \"d997c646-e8fd-4a08-99b6-603ab21587a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh" Nov 03 22:29:06.651008 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:06.650904 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh" Nov 03 22:29:07.051847 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:07.051808 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh"] Nov 03 22:29:07.053556 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:29:07.053518 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd997c646_e8fd_4a08_99b6_603ab21587a7.slice/crio-5492cb2d117d21f0b62d13fedcbef879ecae8ff2272590a41c2b667a6c442583 WatchSource:0}: Error finding container 5492cb2d117d21f0b62d13fedcbef879ecae8ff2272590a41c2b667a6c442583: Status 404 returned error can't find the container with id 5492cb2d117d21f0b62d13fedcbef879ecae8ff2272590a41c2b667a6c442583 Nov 03 22:29:07.515603 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:07.515562 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh" event={"ID":"d997c646-e8fd-4a08-99b6-603ab21587a7","Type":"ContainerStarted","Data":"1cf8491e0ded044ba5a6264da9ac159ca53f0c52b888468b611b1d2bbae3d30e"} Nov 03 22:29:07.515991 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:07.515624 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh" event={"ID":"d997c646-e8fd-4a08-99b6-603ab21587a7","Type":"ContainerStarted","Data":"5492cb2d117d21f0b62d13fedcbef879ecae8ff2272590a41c2b667a6c442583"} Nov 03 22:29:07.636127 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:07.636045 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh" podStartSLOduration=1.636029432 podStartE2EDuration="1.636029432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:07.633914263 +0000 UTC m=+1315.227946234" watchObservedRunningTime="2026-11-03 22:29:07.636029432 +0000 UTC m=+1315.230061439" Nov 03 22:29:07.829682 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:07.829642 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh"] Nov 03 22:29:07.839078 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:07.839023 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh" Nov 03 22:29:07.876326 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:07.876276 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb779fef-d86b-4b16-ad9f-78459edd4e98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e9fef24-xsgwh\" (UID: \"cb779fef-d86b-4b16-ad9f-78459edd4e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh" Nov 03 22:29:07.876527 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:07.876503 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb779fef-d86b-4b16-ad9f-78459edd4e98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e9fef24-xsgwh\" (UID: \"cb779fef-d86b-4b16-ad9f-78459edd4e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh" Nov 03 22:29:07.876724 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:07.876692 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzzfd\" (UniqueName: \"kubernetes.io/projected/cb779fef-d86b-4b16-ad9f-78459edd4e98-kube-api-access-bzzfd\") pod \"maas-api-revoke-sub-8e9fef24-xsgwh\" (UID: \"cb779fef-d86b-4b16-ad9f-78459edd4e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh" Nov 03 22:29:07.887881 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:07.887842 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh"] Nov 03 22:29:07.977725 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:07.977684 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzzfd\" (UniqueName: \"kubernetes.io/projected/cb779fef-d86b-4b16-ad9f-78459edd4e98-kube-api-access-bzzfd\") pod \"maas-api-revoke-sub-8e9fef24-xsgwh\" (UID: \"cb779fef-d86b-4b16-ad9f-78459edd4e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh" Nov 03 22:29:07.977945 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:07.977743 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb779fef-d86b-4b16-ad9f-78459edd4e98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e9fef24-xsgwh\" (UID: \"cb779fef-d86b-4b16-ad9f-78459edd4e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh" Nov 03 22:29:07.977945 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:07.977908 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb779fef-d86b-4b16-ad9f-78459edd4e98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e9fef24-xsgwh\" (UID: \"cb779fef-d86b-4b16-ad9f-78459edd4e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh" Nov 03 22:29:07.978521 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:07.978494 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb779fef-d86b-4b16-ad9f-78459edd4e98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e9fef24-xsgwh\" (UID: \"cb779fef-d86b-4b16-ad9f-78459edd4e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh" Nov 03 22:29:07.980535 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:07.980506 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb779fef-d86b-4b16-ad9f-78459edd4e98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e9fef24-xsgwh\" (UID: \"cb779fef-d86b-4b16-ad9f-78459edd4e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh" Nov 03 22:29:08.038928 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.038895 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzzfd\" (UniqueName: \"kubernetes.io/projected/cb779fef-d86b-4b16-ad9f-78459edd4e98-kube-api-access-bzzfd\") pod \"maas-api-revoke-sub-8e9fef24-xsgwh\" (UID: \"cb779fef-d86b-4b16-ad9f-78459edd4e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh" Nov 03 22:29:08.144997 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.144955 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8"] Nov 03 22:29:08.150221 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.150189 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.158017 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.157980 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh" Nov 03 22:29:08.181037 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.180979 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.181241 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.181052 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.181241 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.181176 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2qss\" (UniqueName: \"kubernetes.io/projected/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-kube-api-access-c2qss\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.181368 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.181311 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.181426 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.181387 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.181483 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.181458 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.181804 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.181579 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.181804 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.181702 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.181804 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.181745 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.249895 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.249824 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8"] Nov 03 22:29:08.283219 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.283180 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.283426 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.283228 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.283426 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.283320 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.283426 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.283353 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.283426 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.283377 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2qss\" (UniqueName: \"kubernetes.io/projected/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-kube-api-access-c2qss\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.283426 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.283424 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.283682 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.283479 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.283682 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.283518 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.283682 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.283632 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.284024 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.284001 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-data\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.284313 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.284293 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-workload-certs\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.284571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.284551 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-workload-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.284937 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.284911 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-credential-socket\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.285144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.285124 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istiod-ca-cert\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.286279 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.286256 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-envoy\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.287665 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.287640 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-podinfo\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.352319 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.352285 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2qss\" (UniqueName: \"kubernetes.io/projected/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-kube-api-access-c2qss\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.352508 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.352351 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-token\") pod \"e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.507841 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.507745 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:08.526486 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.526453 5084 generic.go:356] "Generic (PLEG): container finished" podID="d997c646-e8fd-4a08-99b6-603ab21587a7" containerID="1cf8491e0ded044ba5a6264da9ac159ca53f0c52b888468b611b1d2bbae3d30e" exitCode=0 Nov 03 22:29:08.526931 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.526527 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh" event={"ID":"d997c646-e8fd-4a08-99b6-603ab21587a7","Type":"ContainerDied","Data":"1cf8491e0ded044ba5a6264da9ac159ca53f0c52b888468b611b1d2bbae3d30e"} Nov 03 22:29:08.597431 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.590899 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh"] Nov 03 22:29:08.597523 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:29:08.597469 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb779fef_d86b_4b16_ad9f_78459edd4e98.slice/crio-73f4b4fb1bd7c53d5cfbf24933115af1d7846bf1f7194ca584b333555467f6fe WatchSource:0}: Error finding container 73f4b4fb1bd7c53d5cfbf24933115af1d7846bf1f7194ca584b333555467f6fe: Status 404 returned error can't find the container with id 73f4b4fb1bd7c53d5cfbf24933115af1d7846bf1f7194ca584b333555467f6fe Nov 03 22:29:08.938681 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:29:08.938640 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e61dd6_336d_48cb_bda1_7ee80c1e27b9.slice/crio-bdd809fcff41d11731fcd2acb315a6296ae9eb122647bcbe43767bc229f685e6 WatchSource:0}: Error finding container bdd809fcff41d11731fcd2acb315a6296ae9eb122647bcbe43767bc229f685e6: Status 404 returned error can't find the container with id bdd809fcff41d11731fcd2acb315a6296ae9eb122647bcbe43767bc229f685e6 Nov 03 22:29:08.947167 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.939127 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8"] Nov 03 22:29:08.947167 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.942849 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:29:08.947167 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.942924 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:29:08.947167 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:08.942952 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:29:09.550134 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:09.544508 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" event={"ID":"50e61dd6-336d-48cb-bda1-7ee80c1e27b9","Type":"ContainerStarted","Data":"108f946da24fcfe10d6548ae236e36d49220385fc76ee76ea6626c6ccd567f66"} Nov 03 22:29:09.550134 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:09.544605 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" event={"ID":"50e61dd6-336d-48cb-bda1-7ee80c1e27b9","Type":"ContainerStarted","Data":"bdd809fcff41d11731fcd2acb315a6296ae9eb122647bcbe43767bc229f685e6"} Nov 03 22:29:09.550134 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:09.549957 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh" event={"ID":"cb779fef-d86b-4b16-ad9f-78459edd4e98","Type":"ContainerStarted","Data":"a8c8ebce40a0d5b97422231f26416be8760ac66c96512e3e9c8256e211ee963d"} Nov 03 22:29:09.550134 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:09.550010 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh" event={"ID":"cb779fef-d86b-4b16-ad9f-78459edd4e98","Type":"ContainerStarted","Data":"73f4b4fb1bd7c53d5cfbf24933115af1d7846bf1f7194ca584b333555467f6fe"} Nov 03 22:29:09.887152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:09.879659 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh" podStartSLOduration=2.879635575 podStartE2EDuration="2.879635575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:09.877974041 +0000 UTC m=+1317.472006018" watchObservedRunningTime="2026-11-03 22:29:09.879635575 +0000 UTC m=+1317.473667550" Nov 03 22:29:09.950957 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:09.950852 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" podStartSLOduration=1.9508361239999998 podStartE2EDuration="1.950836124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:09.692713832 +0000 UTC m=+1317.286745807" watchObservedRunningTime="2026-11-03 22:29:09.950836124 +0000 UTC m=+1317.544868099" Nov 03 22:29:10.542092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:10.532148 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:10.817091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:10.816426 5084 generic.go:356] "Generic (PLEG): container finished" podID="cb779fef-d86b-4b16-ad9f-78459edd4e98" containerID="a8c8ebce40a0d5b97422231f26416be8760ac66c96512e3e9c8256e211ee963d" exitCode=0 Nov 03 22:29:10.857092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:10.838261 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh" event={"ID":"cb779fef-d86b-4b16-ad9f-78459edd4e98","Type":"ContainerDied","Data":"a8c8ebce40a0d5b97422231f26416be8760ac66c96512e3e9c8256e211ee963d"} Nov 03 22:29:10.859904 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:10.859865 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:29:10.860210 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:10.860176 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" podUID="50e61dd6-336d-48cb-bda1-7ee80c1e27b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 22:29:11.019403 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:11.019307 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh" Nov 03 22:29:11.204516 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:11.204474 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d997c646-e8fd-4a08-99b6-603ab21587a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d997c646-e8fd-4a08-99b6-603ab21587a7-maas-api-xks-ca\") pod \"d997c646-e8fd-4a08-99b6-603ab21587a7\" (UID: \"d997c646-e8fd-4a08-99b6-603ab21587a7\") " Nov 03 22:29:11.205264 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:11.205237 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d997c646-e8fd-4a08-99b6-603ab21587a7-kube-api-access-5gcqh\" (UniqueName: \"kubernetes.io/projected/d997c646-e8fd-4a08-99b6-603ab21587a7-kube-api-access-5gcqh\") pod \"d997c646-e8fd-4a08-99b6-603ab21587a7\" (UID: \"d997c646-e8fd-4a08-99b6-603ab21587a7\") " Nov 03 22:29:11.212090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:11.205837 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d997c646-e8fd-4a08-99b6-603ab21587a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d997c646-e8fd-4a08-99b6-603ab21587a7-maas-api-openshift-ca\") pod \"d997c646-e8fd-4a08-99b6-603ab21587a7\" (UID: \"d997c646-e8fd-4a08-99b6-603ab21587a7\") " Nov 03 22:29:11.232168 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:11.231317 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d997c646-e8fd-4a08-99b6-603ab21587a7-maas-api-openshift-ca" pod "d997c646-e8fd-4a08-99b6-603ab21587a7" (UID: "d997c646-e8fd-4a08-99b6-603ab21587a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:11.233083 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:11.233032 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d997c646-e8fd-4a08-99b6-603ab21587a7-maas-api-openshift-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:11.249105 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:11.249047 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d997c646-e8fd-4a08-99b6-603ab21587a7-kube-api-access-5gcqh" pod "d997c646-e8fd-4a08-99b6-603ab21587a7" (UID: "d997c646-e8fd-4a08-99b6-603ab21587a7"). InnerVolumeSpecName "kube-api-access-5gcqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:11.249442 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:11.249412 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d997c646-e8fd-4a08-99b6-603ab21587a7-maas-api-xks-ca" pod "d997c646-e8fd-4a08-99b6-603ab21587a7" (UID: "d997c646-e8fd-4a08-99b6-603ab21587a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:11.438203 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:11.425988 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d997c646-e8fd-4a08-99b6-603ab21587a7-maas-api-xks-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:11.495208 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:11.495164 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gcqh\" (UniqueName: \"kubernetes.io/projected/d997c646-e8fd-4a08-99b6-603ab21587a7-kube-api-access-5gcqh\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:11.711024 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:11.710916 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:11.759088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:11.749434 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:29:11.759088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:11.749512 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" podUID="50e61dd6-336d-48cb-bda1-7ee80c1e27b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 22:29:11.951089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:11.948613 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh" Nov 03 22:29:11.980148 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:11.973161 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh" event={"ID":"d997c646-e8fd-4a08-99b6-603ab21587a7","Type":"ContainerDied","Data":"5492cb2d117d21f0b62d13fedcbef879ecae8ff2272590a41c2b667a6c442583"} Nov 03 22:29:11.980148 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:11.973304 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5492cb2d117d21f0b62d13fedcbef879ecae8ff2272590a41c2b667a6c442583" Nov 03 22:29:12.137655 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:12.137598 5084 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.338977 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:12.338922 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.469617 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:12.469261 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:12.469617 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:12.469332 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" podUID="231a9925-fa0d-4a64-b4e6-cdeaf59d44be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.519121 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:12.517334 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:29:12.519121 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:12.517405 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" podUID="50e61dd6-336d-48cb-bda1-7ee80c1e27b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 22:29:12.548956 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:29:12.548913 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:29:12.683362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:12.682413 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.908291 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:12.890354 5084 scope.go:122] "RemoveContainer" containerID="9e2c47f0f893e2d64c1f3427868bebd26932027da0cb3395a577b4031217b11d" Nov 03 22:29:13.152222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:13.152191 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh" Nov 03 22:29:13.312043 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:13.312004 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb779fef-d86b-4b16-ad9f-78459edd4e98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb779fef-d86b-4b16-ad9f-78459edd4e98-maas-api-openshift-ca\") pod \"cb779fef-d86b-4b16-ad9f-78459edd4e98\" (UID: \"cb779fef-d86b-4b16-ad9f-78459edd4e98\") " Nov 03 22:29:13.312445 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:13.312220 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb779fef-d86b-4b16-ad9f-78459edd4e98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb779fef-d86b-4b16-ad9f-78459edd4e98-maas-api-xks-ca\") pod \"cb779fef-d86b-4b16-ad9f-78459edd4e98\" (UID: \"cb779fef-d86b-4b16-ad9f-78459edd4e98\") " Nov 03 22:29:13.312445 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:13.312283 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb779fef-d86b-4b16-ad9f-78459edd4e98-kube-api-access-bzzfd\" (UniqueName: \"kubernetes.io/projected/cb779fef-d86b-4b16-ad9f-78459edd4e98-kube-api-access-bzzfd\") pod \"cb779fef-d86b-4b16-ad9f-78459edd4e98\" (UID: \"cb779fef-d86b-4b16-ad9f-78459edd4e98\") " Nov 03 22:29:13.312695 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:13.312657 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb779fef-d86b-4b16-ad9f-78459edd4e98-maas-api-openshift-ca" pod "cb779fef-d86b-4b16-ad9f-78459edd4e98" (UID: "cb779fef-d86b-4b16-ad9f-78459edd4e98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:13.313014 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:13.312849 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb779fef-d86b-4b16-ad9f-78459edd4e98-maas-api-openshift-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:13.315504 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:13.315465 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb779fef-d86b-4b16-ad9f-78459edd4e98-kube-api-access-bzzfd" pod "cb779fef-d86b-4b16-ad9f-78459edd4e98" (UID: "cb779fef-d86b-4b16-ad9f-78459edd4e98"). InnerVolumeSpecName "kube-api-access-bzzfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:13.315504 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:13.315497 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb779fef-d86b-4b16-ad9f-78459edd4e98-maas-api-xks-ca" pod "cb779fef-d86b-4b16-ad9f-78459edd4e98" (UID: "cb779fef-d86b-4b16-ad9f-78459edd4e98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:13.414040 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:13.413992 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb779fef-d86b-4b16-ad9f-78459edd4e98-maas-api-xks-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:13.414040 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:13.414042 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzzfd\" (UniqueName: \"kubernetes.io/projected/cb779fef-d86b-4b16-ad9f-78459edd4e98-kube-api-access-bzzfd\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:13.509343 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:13.509304 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:29:13.509519 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:13.509365 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" podUID="50e61dd6-336d-48cb-bda1-7ee80c1e27b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 22:29:14.126924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:14.126889 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh" Nov 03 22:29:14.127175 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:14.126890 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh" event={"ID":"cb779fef-d86b-4b16-ad9f-78459edd4e98","Type":"ContainerDied","Data":"73f4b4fb1bd7c53d5cfbf24933115af1d7846bf1f7194ca584b333555467f6fe"} Nov 03 22:29:14.127175 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:14.127048 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73f4b4fb1bd7c53d5cfbf24933115af1d7846bf1f7194ca584b333555467f6fe" Nov 03 22:29:14.509341 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:14.509242 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:29:14.509737 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:14.509328 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" podUID="50e61dd6-336d-48cb-bda1-7ee80c1e27b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 22:29:15.508537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:15.508497 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:29:15.508758 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:15.508568 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" podUID="50e61dd6-336d-48cb-bda1-7ee80c1e27b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 22:29:16.508242 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:16.508203 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:29:16.508675 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:16.508261 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" podUID="50e61dd6-336d-48cb-bda1-7ee80c1e27b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 22:29:17.508635 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:17.508599 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 22:29:17.509058 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:17.508652 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" podUID="50e61dd6-336d-48cb-bda1-7ee80c1e27b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 22:29:18.513272 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:18.513232 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:18.513737 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:18.513538 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:18.514295 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:18.514270 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:29:20.216093 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.216035 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd"] Nov 03 22:29:20.216610 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.216595 5084 cpu_manager.go:414] "containerMap: removing container" podUID="d997c646-e8fd-4a08-99b6-603ab21587a7" containerName="revoke-keys" Nov 03 22:29:20.216658 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.216611 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d997c646-e8fd-4a08-99b6-603ab21587a7" containerName="revoke-keys" Nov 03 22:29:20.216658 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.216642 5084 cpu_manager.go:414] "containerMap: removing container" podUID="cb779fef-d86b-4b16-ad9f-78459edd4e98" containerName="revoke-keys" Nov 03 22:29:20.216658 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.216646 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb779fef-d86b-4b16-ad9f-78459edd4e98" containerName="revoke-keys" Nov 03 22:29:20.216752 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.216721 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="d997c646-e8fd-4a08-99b6-603ab21587a7" containerName="revoke-keys" Nov 03 22:29:20.216752 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.216734 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb779fef-d86b-4b16-ad9f-78459edd4e98" containerName="revoke-keys" Nov 03 22:29:20.220240 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.220217 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd" Nov 03 22:29:20.273219 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.273167 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd"] Nov 03 22:29:20.312716 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.312675 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmmfk\" (UniqueName: \"kubernetes.io/projected/d85c67b7-19ff-4981-8f38-faac63eed31d-kube-api-access-wmmfk\") pod \"maas-api-revoke-sub-b8f0bd85-7t7bd\" (UID: \"d85c67b7-19ff-4981-8f38-faac63eed31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd" Nov 03 22:29:20.312888 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.312831 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85c67b7-19ff-4981-8f38-faac63eed31d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8f0bd85-7t7bd\" (UID: \"d85c67b7-19ff-4981-8f38-faac63eed31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd" Nov 03 22:29:20.312888 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.312858 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85c67b7-19ff-4981-8f38-faac63eed31d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8f0bd85-7t7bd\" (UID: \"d85c67b7-19ff-4981-8f38-faac63eed31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd" Nov 03 22:29:20.319896 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.319862 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:29:20.320149 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.320112 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:29:20.413949 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.413915 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85c67b7-19ff-4981-8f38-faac63eed31d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8f0bd85-7t7bd\" (UID: \"d85c67b7-19ff-4981-8f38-faac63eed31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd" Nov 03 22:29:20.413949 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.413951 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85c67b7-19ff-4981-8f38-faac63eed31d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8f0bd85-7t7bd\" (UID: \"d85c67b7-19ff-4981-8f38-faac63eed31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd" Nov 03 22:29:20.414238 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.414107 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmmfk\" (UniqueName: \"kubernetes.io/projected/d85c67b7-19ff-4981-8f38-faac63eed31d-kube-api-access-wmmfk\") pod \"maas-api-revoke-sub-b8f0bd85-7t7bd\" (UID: \"d85c67b7-19ff-4981-8f38-faac63eed31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd" Nov 03 22:29:20.414742 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.414719 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85c67b7-19ff-4981-8f38-faac63eed31d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8f0bd85-7t7bd\" (UID: \"d85c67b7-19ff-4981-8f38-faac63eed31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd" Nov 03 22:29:20.416494 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.416471 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85c67b7-19ff-4981-8f38-faac63eed31d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8f0bd85-7t7bd\" (UID: \"d85c67b7-19ff-4981-8f38-faac63eed31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd" Nov 03 22:29:20.472932 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.472859 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmmfk\" (UniqueName: \"kubernetes.io/projected/d85c67b7-19ff-4981-8f38-faac63eed31d-kube-api-access-wmmfk\") pod \"maas-api-revoke-sub-b8f0bd85-7t7bd\" (UID: \"d85c67b7-19ff-4981-8f38-faac63eed31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd" Nov 03 22:29:20.535512 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.535469 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd" Nov 03 22:29:20.918444 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:20.918411 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd"] Nov 03 22:29:20.919705 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:29:20.919675 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85c67b7_19ff_4981_8f38_faac63eed31d.slice/crio-e64bf2f98ab170dd60a001553b807423eec2a9cc2f670bf1e3d8a816c319d5b6 WatchSource:0}: Error finding container e64bf2f98ab170dd60a001553b807423eec2a9cc2f670bf1e3d8a816c319d5b6: Status 404 returned error can't find the container with id e64bf2f98ab170dd60a001553b807423eec2a9cc2f670bf1e3d8a816c319d5b6 Nov 03 22:29:21.184202 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:21.184164 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd" event={"ID":"d85c67b7-19ff-4981-8f38-faac63eed31d","Type":"ContainerStarted","Data":"e64bf2f98ab170dd60a001553b807423eec2a9cc2f670bf1e3d8a816c319d5b6"} Nov 03 22:29:21.921464 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:21.921428 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw"] Nov 03 22:29:21.926500 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:21.926469 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.035241 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.035196 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7vdb\" (UniqueName: \"kubernetes.io/projected/c05a9c50-a87e-4fa7-b3d7-c06557223767-kube-api-access-s7vdb\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.035456 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.035426 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.035608 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.035583 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.035660 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.035626 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c05a9c50-a87e-4fa7-b3d7-c06557223767-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.035709 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.035686 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.035757 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.035717 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.035815 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.035755 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.035896 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.035870 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.036303 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.036077 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.036932 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.036909 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw"] Nov 03 22:29:22.137532 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.137488 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.137532 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.137533 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.137772 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.137587 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.137772 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.137628 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7vdb\" (UniqueName: \"kubernetes.io/projected/c05a9c50-a87e-4fa7-b3d7-c06557223767-kube-api-access-s7vdb\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.137772 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.137731 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.138103 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.137940 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.138103 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.137987 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c05a9c50-a87e-4fa7-b3d7-c06557223767-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.138103 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.137998 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.138103 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.138019 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.138380 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.138103 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.138380 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.138306 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.138528 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.138499 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.138707 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.138687 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.138840 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.138815 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c05a9c50-a87e-4fa7-b3d7-c06557223767-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.140471 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.140448 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.140561 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.140478 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.193790 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.193704 5084 generic.go:356] "Generic (PLEG): container finished" podID="d85c67b7-19ff-4981-8f38-faac63eed31d" containerID="157889b2a5bcdb2edaaee2205de38e62bed07c78d14ea89cf8380dbafaabddc5" exitCode=0 Nov 03 22:29:22.194023 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.193802 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd" event={"ID":"d85c67b7-19ff-4981-8f38-faac63eed31d","Type":"ContainerDied","Data":"157889b2a5bcdb2edaaee2205de38e62bed07c78d14ea89cf8380dbafaabddc5"} Nov 03 22:29:22.196263 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.196230 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.196393 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.196349 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7vdb\" (UniqueName: \"kubernetes.io/projected/c05a9c50-a87e-4fa7-b3d7-c06557223767-kube-api-access-s7vdb\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.248907 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.248865 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:22.688844 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.688805 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw"] Nov 03 22:29:22.689903 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:29:22.689869 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05a9c50_a87e_4fa7_b3d7_c06557223767.slice/crio-c768ab936a1064a54e901ac19a34a961bb6b8827a654a8b321f1f7fe523963ab WatchSource:0}: Error finding container c768ab936a1064a54e901ac19a34a961bb6b8827a654a8b321f1f7fe523963ab: Status 404 returned error can't find the container with id c768ab936a1064a54e901ac19a34a961bb6b8827a654a8b321f1f7fe523963ab Nov 03 22:29:22.692829 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.692765 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:29:22.692956 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.692887 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:29:22.692956 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:22.692936 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:29:23.204303 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:23.204261 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" event={"ID":"c05a9c50-a87e-4fa7-b3d7-c06557223767","Type":"ContainerStarted","Data":"6b6af7f7a43bfc14fe1e79676dd7959217a4de781bb17c20fbe8d56f44f5498e"} Nov 03 22:29:23.204829 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:23.204322 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" event={"ID":"c05a9c50-a87e-4fa7-b3d7-c06557223767","Type":"ContainerStarted","Data":"c768ab936a1064a54e901ac19a34a961bb6b8827a654a8b321f1f7fe523963ab"} Nov 03 22:29:23.251464 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:23.251417 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:23.252863 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:23.252827 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:29:23.253073 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:23.252889 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" podUID="c05a9c50-a87e-4fa7-b3d7-c06557223767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 22:29:23.331437 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:23.330881 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" podStartSLOduration=2.330863669 podStartE2EDuration="2.330863669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:23.328592118 +0000 UTC m=+1330.922624097" watchObservedRunningTime="2026-11-03 22:29:23.330863669 +0000 UTC m=+1330.924895642" Nov 03 22:29:23.436977 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:23.436940 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd" Nov 03 22:29:23.586769 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:23.586724 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d85c67b7-19ff-4981-8f38-faac63eed31d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85c67b7-19ff-4981-8f38-faac63eed31d-maas-api-openshift-ca\") pod \"d85c67b7-19ff-4981-8f38-faac63eed31d\" (UID: \"d85c67b7-19ff-4981-8f38-faac63eed31d\") " Nov 03 22:29:23.587002 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:23.586829 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d85c67b7-19ff-4981-8f38-faac63eed31d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85c67b7-19ff-4981-8f38-faac63eed31d-maas-api-xks-ca\") pod \"d85c67b7-19ff-4981-8f38-faac63eed31d\" (UID: \"d85c67b7-19ff-4981-8f38-faac63eed31d\") " Nov 03 22:29:23.587002 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:23.586882 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85c67b7-19ff-4981-8f38-faac63eed31d-kube-api-access-wmmfk\" (UniqueName: \"kubernetes.io/projected/d85c67b7-19ff-4981-8f38-faac63eed31d-kube-api-access-wmmfk\") pod \"d85c67b7-19ff-4981-8f38-faac63eed31d\" (UID: \"d85c67b7-19ff-4981-8f38-faac63eed31d\") " Nov 03 22:29:23.587880 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:23.587840 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85c67b7-19ff-4981-8f38-faac63eed31d-maas-api-openshift-ca" pod "d85c67b7-19ff-4981-8f38-faac63eed31d" (UID: "d85c67b7-19ff-4981-8f38-faac63eed31d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:23.590202 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:23.590167 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d85c67b7-19ff-4981-8f38-faac63eed31d-maas-api-xks-ca" pod "d85c67b7-19ff-4981-8f38-faac63eed31d" (UID: "d85c67b7-19ff-4981-8f38-faac63eed31d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:23.590358 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:23.590311 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85c67b7-19ff-4981-8f38-faac63eed31d-kube-api-access-wmmfk" pod "d85c67b7-19ff-4981-8f38-faac63eed31d" (UID: "d85c67b7-19ff-4981-8f38-faac63eed31d"). InnerVolumeSpecName "kube-api-access-wmmfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:23.687931 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:23.687883 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmmfk\" (UniqueName: \"kubernetes.io/projected/d85c67b7-19ff-4981-8f38-faac63eed31d-kube-api-access-wmmfk\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:23.687931 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:23.687919 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85c67b7-19ff-4981-8f38-faac63eed31d-maas-api-openshift-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:23.687931 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:23.687929 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85c67b7-19ff-4981-8f38-faac63eed31d-maas-api-xks-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:24.214606 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:24.214563 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd" Nov 03 22:29:24.214606 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:24.214578 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd" event={"ID":"d85c67b7-19ff-4981-8f38-faac63eed31d","Type":"ContainerDied","Data":"e64bf2f98ab170dd60a001553b807423eec2a9cc2f670bf1e3d8a816c319d5b6"} Nov 03 22:29:24.215229 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:24.214658 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e64bf2f98ab170dd60a001553b807423eec2a9cc2f670bf1e3d8a816c319d5b6" Nov 03 22:29:24.249809 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:24.249758 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:29:24.249998 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:24.249837 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" podUID="c05a9c50-a87e-4fa7-b3d7-c06557223767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 22:29:24.813879 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:24.813830 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk"] Nov 03 22:29:24.814530 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:24.814508 5084 cpu_manager.go:414] "containerMap: removing container" podUID="d85c67b7-19ff-4981-8f38-faac63eed31d" containerName="revoke-keys" Nov 03 22:29:24.814646 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:24.814531 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85c67b7-19ff-4981-8f38-faac63eed31d" containerName="revoke-keys" Nov 03 22:29:24.814703 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:24.814680 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85c67b7-19ff-4981-8f38-faac63eed31d" containerName="revoke-keys" Nov 03 22:29:24.817872 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:24.817845 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk" Nov 03 22:29:24.880087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:24.872076 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:29:24.902916 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:24.902872 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk"] Nov 03 22:29:24.903537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:24.903226 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:29:25.004865 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:25.004817 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa23f164-97be-463e-9dd1-1dbae8a0b59c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9527ca05-8lkqk\" (UID: \"fa23f164-97be-463e-9dd1-1dbae8a0b59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk" Nov 03 22:29:25.005107 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:25.004964 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncsft\" (UniqueName: \"kubernetes.io/projected/fa23f164-97be-463e-9dd1-1dbae8a0b59c-kube-api-access-ncsft\") pod \"maas-api-revoke-sub-9527ca05-8lkqk\" (UID: \"fa23f164-97be-463e-9dd1-1dbae8a0b59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk" Nov 03 22:29:25.005107 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:25.005100 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa23f164-97be-463e-9dd1-1dbae8a0b59c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9527ca05-8lkqk\" (UID: \"fa23f164-97be-463e-9dd1-1dbae8a0b59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk" Nov 03 22:29:25.106809 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:25.106711 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa23f164-97be-463e-9dd1-1dbae8a0b59c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9527ca05-8lkqk\" (UID: \"fa23f164-97be-463e-9dd1-1dbae8a0b59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk" Nov 03 22:29:25.106809 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:25.106779 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncsft\" (UniqueName: \"kubernetes.io/projected/fa23f164-97be-463e-9dd1-1dbae8a0b59c-kube-api-access-ncsft\") pod \"maas-api-revoke-sub-9527ca05-8lkqk\" (UID: \"fa23f164-97be-463e-9dd1-1dbae8a0b59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk" Nov 03 22:29:25.107049 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:25.106836 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa23f164-97be-463e-9dd1-1dbae8a0b59c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9527ca05-8lkqk\" (UID: \"fa23f164-97be-463e-9dd1-1dbae8a0b59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk" Nov 03 22:29:25.107676 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:25.107645 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa23f164-97be-463e-9dd1-1dbae8a0b59c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9527ca05-8lkqk\" (UID: \"fa23f164-97be-463e-9dd1-1dbae8a0b59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk" Nov 03 22:29:25.109551 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:25.109516 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa23f164-97be-463e-9dd1-1dbae8a0b59c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9527ca05-8lkqk\" (UID: \"fa23f164-97be-463e-9dd1-1dbae8a0b59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk" Nov 03 22:29:25.194086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:25.194021 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncsft\" (UniqueName: \"kubernetes.io/projected/fa23f164-97be-463e-9dd1-1dbae8a0b59c-kube-api-access-ncsft\") pod \"maas-api-revoke-sub-9527ca05-8lkqk\" (UID: \"fa23f164-97be-463e-9dd1-1dbae8a0b59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk" Nov 03 22:29:25.250281 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:25.250244 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:29:25.250672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:25.250303 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" podUID="c05a9c50-a87e-4fa7-b3d7-c06557223767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 22:29:25.432592 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:25.432488 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk" Nov 03 22:29:25.852556 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:25.852520 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk"] Nov 03 22:29:25.853629 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:29:25.853576 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa23f164_97be_463e_9dd1_1dbae8a0b59c.slice/crio-5f8d7781a8705caf7637854c89681c4bc32800a493d86561692279aa6a4e70d9 WatchSource:0}: Error finding container 5f8d7781a8705caf7637854c89681c4bc32800a493d86561692279aa6a4e70d9: Status 404 returned error can't find the container with id 5f8d7781a8705caf7637854c89681c4bc32800a493d86561692279aa6a4e70d9 Nov 03 22:29:26.230515 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:26.230474 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk" event={"ID":"fa23f164-97be-463e-9dd1-1dbae8a0b59c","Type":"ContainerStarted","Data":"5f8d7781a8705caf7637854c89681c4bc32800a493d86561692279aa6a4e70d9"} Nov 03 22:29:26.250213 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:26.250171 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:29:26.250443 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:26.250258 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" podUID="c05a9c50-a87e-4fa7-b3d7-c06557223767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 22:29:27.239716 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:27.239682 5084 generic.go:356] "Generic (PLEG): container finished" podID="fa23f164-97be-463e-9dd1-1dbae8a0b59c" containerID="7d8d0f28fabd04272c8a29893d6a14b1d7bc2df80207fcc7b9d6f59166bcea59" exitCode=0 Nov 03 22:29:27.239912 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:27.239768 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk" event={"ID":"fa23f164-97be-463e-9dd1-1dbae8a0b59c","Type":"ContainerDied","Data":"7d8d0f28fabd04272c8a29893d6a14b1d7bc2df80207fcc7b9d6f59166bcea59"} Nov 03 22:29:27.250050 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:27.250018 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:29:27.250196 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:27.250100 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" podUID="c05a9c50-a87e-4fa7-b3d7-c06557223767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 22:29:28.249865 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:28.249833 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:29:28.250207 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:28.249897 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" podUID="c05a9c50-a87e-4fa7-b3d7-c06557223767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 22:29:28.531912 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:28.531882 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk" Nov 03 22:29:28.601932 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:28.601881 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa23f164-97be-463e-9dd1-1dbae8a0b59c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa23f164-97be-463e-9dd1-1dbae8a0b59c-maas-api-xks-ca\") pod \"fa23f164-97be-463e-9dd1-1dbae8a0b59c\" (UID: \"fa23f164-97be-463e-9dd1-1dbae8a0b59c\") " Nov 03 22:29:28.602191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:28.602051 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa23f164-97be-463e-9dd1-1dbae8a0b59c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa23f164-97be-463e-9dd1-1dbae8a0b59c-maas-api-openshift-ca\") pod \"fa23f164-97be-463e-9dd1-1dbae8a0b59c\" (UID: \"fa23f164-97be-463e-9dd1-1dbae8a0b59c\") " Nov 03 22:29:28.602191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:28.602146 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa23f164-97be-463e-9dd1-1dbae8a0b59c-kube-api-access-ncsft\" (UniqueName: \"kubernetes.io/projected/fa23f164-97be-463e-9dd1-1dbae8a0b59c-kube-api-access-ncsft\") pod \"fa23f164-97be-463e-9dd1-1dbae8a0b59c\" (UID: \"fa23f164-97be-463e-9dd1-1dbae8a0b59c\") " Nov 03 22:29:28.602433 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:28.602394 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa23f164-97be-463e-9dd1-1dbae8a0b59c-maas-api-openshift-ca" pod "fa23f164-97be-463e-9dd1-1dbae8a0b59c" (UID: "fa23f164-97be-463e-9dd1-1dbae8a0b59c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:28.602657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:28.602638 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa23f164-97be-463e-9dd1-1dbae8a0b59c-maas-api-openshift-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:28.604395 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:28.604367 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa23f164-97be-463e-9dd1-1dbae8a0b59c-maas-api-xks-ca" pod "fa23f164-97be-463e-9dd1-1dbae8a0b59c" (UID: "fa23f164-97be-463e-9dd1-1dbae8a0b59c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:28.604753 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:28.604727 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa23f164-97be-463e-9dd1-1dbae8a0b59c-kube-api-access-ncsft" pod "fa23f164-97be-463e-9dd1-1dbae8a0b59c" (UID: "fa23f164-97be-463e-9dd1-1dbae8a0b59c"). InnerVolumeSpecName "kube-api-access-ncsft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:28.703482 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:28.703445 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa23f164-97be-463e-9dd1-1dbae8a0b59c-maas-api-xks-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:28.703482 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:28.703479 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncsft\" (UniqueName: \"kubernetes.io/projected/fa23f164-97be-463e-9dd1-1dbae8a0b59c-kube-api-access-ncsft\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:29.249326 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:29.249288 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:29:29.249532 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:29.249348 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" podUID="c05a9c50-a87e-4fa7-b3d7-c06557223767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 22:29:29.257220 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:29.257186 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk" Nov 03 22:29:29.257220 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:29.257198 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk" event={"ID":"fa23f164-97be-463e-9dd1-1dbae8a0b59c","Type":"ContainerDied","Data":"5f8d7781a8705caf7637854c89681c4bc32800a493d86561692279aa6a4e70d9"} Nov 03 22:29:29.257807 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:29.257245 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f8d7781a8705caf7637854c89681c4bc32800a493d86561692279aa6a4e70d9" Nov 03 22:29:30.254074 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:30.254033 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:30.254464 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:30.254444 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:30.255216 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:30.255191 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:29:36.175090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.174897 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d"] Nov 03 22:29:36.196019 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.195976 5084 cpu_manager.go:414] "containerMap: removing container" podUID="fa23f164-97be-463e-9dd1-1dbae8a0b59c" containerName="revoke-keys" Nov 03 22:29:36.196302 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.196268 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa23f164-97be-463e-9dd1-1dbae8a0b59c" containerName="revoke-keys" Nov 03 22:29:36.196673 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.196654 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa23f164-97be-463e-9dd1-1dbae8a0b59c" containerName="revoke-keys" Nov 03 22:29:36.235103 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.228165 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.348119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.324474 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.348119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.324546 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.348119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.324605 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.348119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.324641 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.348119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.324700 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgclh\" (UniqueName: \"kubernetes.io/projected/462d5bbd-4b7b-4c70-857a-99964a7bdef3-kube-api-access-xgclh\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.348119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.324730 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.348119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.324816 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.348119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.324841 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.348119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.324889 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.427153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.361995 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d"] Nov 03 22:29:36.508786 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.466582 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgclh\" (UniqueName: \"kubernetes.io/projected/462d5bbd-4b7b-4c70-857a-99964a7bdef3-kube-api-access-xgclh\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.742397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.742292 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.780367 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.780316 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.780652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.780628 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.780804 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.780789 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.784531 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.784489 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.784809 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.784787 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.785000 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.784982 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.785170 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.785149 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.785904 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.620398 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r"] Nov 03 22:29:36.799088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.652213 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 22:29:36.799088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.677276 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:36.863090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.836037 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r"] Nov 03 22:29:36.863090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.857532 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:36.888463 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.888410 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:36.918335 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.918040 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.918335 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.918282 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.919250 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.919224 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgclh\" (UniqueName: \"kubernetes.io/projected/462d5bbd-4b7b-4c70-857a-99964a7bdef3-kube-api-access-xgclh\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.919582 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.919560 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.919885 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.919867 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.931544 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.931495 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:36.962108 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:36.959902 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:37.034139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.024049 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.034139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.024127 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4324c92c-c67b-4407-868b-eecc823ac436-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.034139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.024202 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4324c92c-c67b-4407-868b-eecc823ac436-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.034139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.024255 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.034139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.024284 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.034139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.024341 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.034139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.024410 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.034139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.024514 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4324c92c-c67b-4407-868b-eecc823ac436-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.034139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.024541 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk7jj\" (UniqueName: \"kubernetes.io/projected/4324c92c-c67b-4407-868b-eecc823ac436-kube-api-access-kk7jj\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.210796 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.116993 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" podUID="3b3927af-60a2-4432-99d6-b999a96f315f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:37.210796 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.150842 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.211391 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.210882 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4324c92c-c67b-4407-868b-eecc823ac436-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.211391 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.210989 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4324c92c-c67b-4407-868b-eecc823ac436-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.211391 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.211093 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.211391 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.211124 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.211391 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.211194 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.211391 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.211303 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.211726 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.211444 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4324c92c-c67b-4407-868b-eecc823ac436-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.211726 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.211475 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7jj\" (UniqueName: \"kubernetes.io/projected/4324c92c-c67b-4407-868b-eecc823ac436-kube-api-access-kk7jj\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.211869 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.151395 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-credential-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.243560 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.212705 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4324c92c-c67b-4407-868b-eecc823ac436-istiod-ca-cert\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.243560 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.169108 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:37.243560 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.213616 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-istio-data\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.245096 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.244516 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-workload-certs\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.267181 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.253412 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 03 22:29:37.299698 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.299592 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-workload-socket\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.302423 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.302389 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-istio-envoy\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.468135 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.303032 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4324c92c-c67b-4407-868b-eecc823ac436-istio-podinfo\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.468443 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.316806 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4324c92c-c67b-4407-868b-eecc823ac436-istio-token\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.468556 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.349874 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:37.540113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.514593 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk7jj\" (UniqueName: \"kubernetes.io/projected/4324c92c-c67b-4407-868b-eecc823ac436-kube-api-access-kk7jj\") pod \"e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:37.618168 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.569647 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:37.813103 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.794972 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:37.813103 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.795654 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:29:37.813103 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:37.812820 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:38.296089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:38.294224 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 03 22:29:38.319222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:38.305504 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:29:38.624590 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:38.624481 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:38.801220 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:38.801176 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:38.958290 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:38.958178 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:38.997990 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:38.996776 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:29:39.403053 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:39.402989 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" podUID="3b3927af-60a2-4432-99d6-b999a96f315f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:39.425724 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:39.424190 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:29:39.799803 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:39.799766 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r"] Nov 03 22:29:39.804581 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:29:39.804544 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4324c92c_c67b_4407_868b_eecc823ac436.slice/crio-318570719d14aa0bd8cce7c36f296f556b606e4ac2648a72e18259bd8ebd47b2 WatchSource:0}: Error finding container 318570719d14aa0bd8cce7c36f296f556b606e4ac2648a72e18259bd8ebd47b2: Status 404 returned error can't find the container with id 318570719d14aa0bd8cce7c36f296f556b606e4ac2648a72e18259bd8ebd47b2 Nov 03 22:29:39.808467 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:39.808414 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:29:39.808607 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:39.808519 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:29:39.808607 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:39.808564 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:29:39.938298 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:39.938265 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d"] Nov 03 22:29:39.939358 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:39.939285 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" event={"ID":"4324c92c-c67b-4407-868b-eecc823ac436","Type":"ContainerStarted","Data":"114c46a3f6d4e6e3ea4a47e0ce33fffea4cb55a37f860342ad2184b37fec5189"} Nov 03 22:29:39.939358 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:39.939352 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" event={"ID":"4324c92c-c67b-4407-868b-eecc823ac436","Type":"ContainerStarted","Data":"318570719d14aa0bd8cce7c36f296f556b606e4ac2648a72e18259bd8ebd47b2"} Nov 03 22:29:39.949189 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:39.949035 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:29:39.949189 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:39.949153 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:29:39.950246 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:39.949198 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:29:40.072000 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:40.071224 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" podStartSLOduration=4.071134927 podStartE2EDuration="4.071134927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:40.064630785 +0000 UTC m=+1347.658662760" watchObservedRunningTime="2026-11-03 22:29:40.071134927 +0000 UTC m=+1347.665167337" Nov 03 22:29:40.822286 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:40.822234 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:40.826754 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:40.826708 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:29:40.826932 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:40.826778 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" podUID="4324c92c-c67b-4407-868b-eecc823ac436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 22:29:40.851686 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:40.851631 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts"] Nov 03 22:29:40.864420 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:40.864385 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts" Nov 03 22:29:40.891219 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:40.890895 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8snj\" (UniqueName: \"kubernetes.io/projected/865851d3-e8f5-40de-8770-34ecfafdf219-kube-api-access-q8snj\") pod \"maas-api-revoke-sub-5c5a1324-6cvts\" (UID: \"865851d3-e8f5-40de-8770-34ecfafdf219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts" Nov 03 22:29:40.891219 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:40.891151 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865851d3-e8f5-40de-8770-34ecfafdf219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c5a1324-6cvts\" (UID: \"865851d3-e8f5-40de-8770-34ecfafdf219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts" Nov 03 22:29:40.891219 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:40.891202 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865851d3-e8f5-40de-8770-34ecfafdf219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c5a1324-6cvts\" (UID: \"865851d3-e8f5-40de-8770-34ecfafdf219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts" Nov 03 22:29:40.944144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:40.944110 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:29:40.986090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:40.949536 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts"] Nov 03 22:29:41.008105 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:40.996899 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865851d3-e8f5-40de-8770-34ecfafdf219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c5a1324-6cvts\" (UID: \"865851d3-e8f5-40de-8770-34ecfafdf219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts" Nov 03 22:29:41.013094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:41.008443 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865851d3-e8f5-40de-8770-34ecfafdf219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c5a1324-6cvts\" (UID: \"865851d3-e8f5-40de-8770-34ecfafdf219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts" Nov 03 22:29:41.013562 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:41.013524 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8snj\" (UniqueName: \"kubernetes.io/projected/865851d3-e8f5-40de-8770-34ecfafdf219-kube-api-access-q8snj\") pod \"maas-api-revoke-sub-5c5a1324-6cvts\" (UID: \"865851d3-e8f5-40de-8770-34ecfafdf219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts" Nov 03 22:29:41.035394 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:41.005482 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:29:41.035394 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:41.035215 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865851d3-e8f5-40de-8770-34ecfafdf219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c5a1324-6cvts\" (UID: \"865851d3-e8f5-40de-8770-34ecfafdf219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts" Nov 03 22:29:41.039177 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:41.039115 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865851d3-e8f5-40de-8770-34ecfafdf219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c5a1324-6cvts\" (UID: \"865851d3-e8f5-40de-8770-34ecfafdf219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts" Nov 03 22:29:41.052107 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:41.049743 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" event={"ID":"462d5bbd-4b7b-4c70-857a-99964a7bdef3","Type":"ContainerStarted","Data":"f1fd7970e135d1b999ae70429acb4defb6bf308e838d1a88b34240e110786151"} Nov 03 22:29:41.052107 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:41.049828 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" event={"ID":"462d5bbd-4b7b-4c70-857a-99964a7bdef3","Type":"ContainerStarted","Data":"228c44b501e112ea85fa9d70508226f23e88d9eb9962fa85b7f127d9e51be0ad"} Nov 03 22:29:41.147381 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:41.147301 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8snj\" (UniqueName: \"kubernetes.io/projected/865851d3-e8f5-40de-8770-34ecfafdf219-kube-api-access-q8snj\") pod \"maas-api-revoke-sub-5c5a1324-6cvts\" (UID: \"865851d3-e8f5-40de-8770-34ecfafdf219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts" Nov 03 22:29:41.229158 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:41.225650 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" podStartSLOduration=5.215052395 podStartE2EDuration="5.215052395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:41.180514937 +0000 UTC m=+1348.774546912" watchObservedRunningTime="2026-11-03 22:29:41.215052395 +0000 UTC m=+1348.809084370" Nov 03 22:29:41.337487 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:41.337444 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts" Nov 03 22:29:41.601149 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:41.595136 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:41.601149 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:41.597605 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 22:29:41.601149 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:41.597666 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" podUID="462d5bbd-4b7b-4c70-857a-99964a7bdef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 22:29:41.827836 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:41.827743 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:29:41.827836 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:41.827802 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" podUID="4324c92c-c67b-4407-868b-eecc823ac436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 22:29:41.929430 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:29:41.929395 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865851d3_e8f5_40de_8770_34ecfafdf219.slice/crio-5bfb7ca7dd72abcb03858106b79864eb3006bddfcdc044256918a69313959c27 WatchSource:0}: Error finding container 5bfb7ca7dd72abcb03858106b79864eb3006bddfcdc044256918a69313959c27: Status 404 returned error can't find the container with id 5bfb7ca7dd72abcb03858106b79864eb3006bddfcdc044256918a69313959c27 Nov 03 22:29:41.934430 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:41.934399 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts"] Nov 03 22:29:42.111415 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:29:42.097136 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:29:42.111415 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:42.099685 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts" event={"ID":"865851d3-e8f5-40de-8770-34ecfafdf219","Type":"ContainerStarted","Data":"5bfb7ca7dd72abcb03858106b79864eb3006bddfcdc044256918a69313959c27"} Nov 03 22:29:42.582320 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:42.582282 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 22:29:42.582600 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:42.582572 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" podUID="462d5bbd-4b7b-4c70-857a-99964a7bdef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 22:29:42.822754 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:42.822716 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:29:42.823124 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:42.823045 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" podUID="4324c92c-c67b-4407-868b-eecc823ac436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 22:29:43.126820 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:43.126718 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts" event={"ID":"865851d3-e8f5-40de-8770-34ecfafdf219","Type":"ContainerStarted","Data":"cbad2739761b20e7b4f4847cc365575d3e4af0660ef27d29a6fb6dbeea1b3cee"} Nov 03 22:29:43.252155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:43.252089 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts" podStartSLOduration=3.252072379 podStartE2EDuration="3.252072379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:43.246603672 +0000 UTC m=+1350.840635669" watchObservedRunningTime="2026-11-03 22:29:43.252072379 +0000 UTC m=+1350.846104370" Nov 03 22:29:43.602090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:43.583521 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 22:29:43.602090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:43.583591 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" podUID="462d5bbd-4b7b-4c70-857a-99964a7bdef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 22:29:43.820604 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:43.820559 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:29:43.820792 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:43.820617 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" podUID="4324c92c-c67b-4407-868b-eecc823ac436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 22:29:44.136478 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:44.136440 5084 generic.go:356] "Generic (PLEG): container finished" podID="865851d3-e8f5-40de-8770-34ecfafdf219" containerID="cbad2739761b20e7b4f4847cc365575d3e4af0660ef27d29a6fb6dbeea1b3cee" exitCode=0 Nov 03 22:29:44.137102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:44.136576 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts" event={"ID":"865851d3-e8f5-40de-8770-34ecfafdf219","Type":"ContainerDied","Data":"cbad2739761b20e7b4f4847cc365575d3e4af0660ef27d29a6fb6dbeea1b3cee"} Nov 03 22:29:44.582767 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:44.581494 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 22:29:44.582767 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:44.581558 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" podUID="462d5bbd-4b7b-4c70-857a-99964a7bdef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 22:29:44.822147 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:44.821563 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:29:44.822147 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:44.821633 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" podUID="4324c92c-c67b-4407-868b-eecc823ac436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 22:29:45.619759 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:45.619721 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 22:29:45.620419 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:45.620364 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" podUID="462d5bbd-4b7b-4c70-857a-99964a7bdef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 22:29:45.625174 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:45.623804 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts" Nov 03 22:29:45.650531 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:45.650493 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865851d3-e8f5-40de-8770-34ecfafdf219-kube-api-access-q8snj\" (UniqueName: \"kubernetes.io/projected/865851d3-e8f5-40de-8770-34ecfafdf219-kube-api-access-q8snj\") pod \"865851d3-e8f5-40de-8770-34ecfafdf219\" (UID: \"865851d3-e8f5-40de-8770-34ecfafdf219\") " Nov 03 22:29:45.650694 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:45.650626 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/865851d3-e8f5-40de-8770-34ecfafdf219-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865851d3-e8f5-40de-8770-34ecfafdf219-maas-api-xks-ca\") pod \"865851d3-e8f5-40de-8770-34ecfafdf219\" (UID: \"865851d3-e8f5-40de-8770-34ecfafdf219\") " Nov 03 22:29:45.650694 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:45.650676 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/865851d3-e8f5-40de-8770-34ecfafdf219-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865851d3-e8f5-40de-8770-34ecfafdf219-maas-api-openshift-ca\") pod \"865851d3-e8f5-40de-8770-34ecfafdf219\" (UID: \"865851d3-e8f5-40de-8770-34ecfafdf219\") " Nov 03 22:29:45.651041 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:45.651016 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/865851d3-e8f5-40de-8770-34ecfafdf219-maas-api-openshift-ca" pod "865851d3-e8f5-40de-8770-34ecfafdf219" (UID: "865851d3-e8f5-40de-8770-34ecfafdf219"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:45.651216 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:45.651199 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865851d3-e8f5-40de-8770-34ecfafdf219-maas-api-openshift-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:45.652778 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:45.652751 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/865851d3-e8f5-40de-8770-34ecfafdf219-maas-api-xks-ca" pod "865851d3-e8f5-40de-8770-34ecfafdf219" (UID: "865851d3-e8f5-40de-8770-34ecfafdf219"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:45.652896 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:45.652866 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865851d3-e8f5-40de-8770-34ecfafdf219-kube-api-access-q8snj" pod "865851d3-e8f5-40de-8770-34ecfafdf219" (UID: "865851d3-e8f5-40de-8770-34ecfafdf219"). InnerVolumeSpecName "kube-api-access-q8snj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:45.778152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:45.752697 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8snj\" (UniqueName: \"kubernetes.io/projected/865851d3-e8f5-40de-8770-34ecfafdf219-kube-api-access-q8snj\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:45.778152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:45.752729 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865851d3-e8f5-40de-8770-34ecfafdf219-maas-api-xks-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:45.832902 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:45.832426 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:29:45.832902 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:45.832492 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" podUID="4324c92c-c67b-4407-868b-eecc823ac436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 22:29:46.176149 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.175736 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts" event={"ID":"865851d3-e8f5-40de-8770-34ecfafdf219","Type":"ContainerDied","Data":"5bfb7ca7dd72abcb03858106b79864eb3006bddfcdc044256918a69313959c27"} Nov 03 22:29:46.176149 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.175809 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bfb7ca7dd72abcb03858106b79864eb3006bddfcdc044256918a69313959c27" Nov 03 22:29:46.176149 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.175902 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts" Nov 03 22:29:46.514342 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.514236 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg"] Nov 03 22:29:46.515581 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.515547 5084 cpu_manager.go:414] "containerMap: removing container" podUID="865851d3-e8f5-40de-8770-34ecfafdf219" containerName="revoke-keys" Nov 03 22:29:46.515760 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.515747 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865851d3-e8f5-40de-8770-34ecfafdf219" containerName="revoke-keys" Nov 03 22:29:46.516020 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.516007 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="865851d3-e8f5-40de-8770-34ecfafdf219" containerName="revoke-keys" Nov 03 22:29:46.528472 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.528436 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg" Nov 03 22:29:46.576105 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.555668 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg"] Nov 03 22:29:46.586089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.583438 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:29:46.586089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.584168 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:29:46.591096 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.588513 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 22:29:46.591096 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.588574 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" podUID="462d5bbd-4b7b-4c70-857a-99964a7bdef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 22:29:46.652116 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.649680 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvtg2\" (UniqueName: \"kubernetes.io/projected/70844697-f6c3-4f12-8fe3-c0d80d501784-kube-api-access-tvtg2\") pod \"maas-api-revoke-sub-950f43c1-x65wg\" (UID: \"70844697-f6c3-4f12-8fe3-c0d80d501784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg" Nov 03 22:29:46.652116 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.649771 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70844697-f6c3-4f12-8fe3-c0d80d501784-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-950f43c1-x65wg\" (UID: \"70844697-f6c3-4f12-8fe3-c0d80d501784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg" Nov 03 22:29:46.652116 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.649889 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70844697-f6c3-4f12-8fe3-c0d80d501784-maas-api-xks-ca\") pod \"maas-api-revoke-sub-950f43c1-x65wg\" (UID: \"70844697-f6c3-4f12-8fe3-c0d80d501784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg" Nov 03 22:29:46.751057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.751007 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtg2\" (UniqueName: \"kubernetes.io/projected/70844697-f6c3-4f12-8fe3-c0d80d501784-kube-api-access-tvtg2\") pod \"maas-api-revoke-sub-950f43c1-x65wg\" (UID: \"70844697-f6c3-4f12-8fe3-c0d80d501784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg" Nov 03 22:29:46.751271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.751108 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70844697-f6c3-4f12-8fe3-c0d80d501784-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-950f43c1-x65wg\" (UID: \"70844697-f6c3-4f12-8fe3-c0d80d501784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg" Nov 03 22:29:46.751881 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.751855 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70844697-f6c3-4f12-8fe3-c0d80d501784-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-950f43c1-x65wg\" (UID: \"70844697-f6c3-4f12-8fe3-c0d80d501784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg" Nov 03 22:29:46.752023 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.751205 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70844697-f6c3-4f12-8fe3-c0d80d501784-maas-api-xks-ca\") pod \"maas-api-revoke-sub-950f43c1-x65wg\" (UID: \"70844697-f6c3-4f12-8fe3-c0d80d501784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg" Nov 03 22:29:46.753896 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.753870 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70844697-f6c3-4f12-8fe3-c0d80d501784-maas-api-xks-ca\") pod \"maas-api-revoke-sub-950f43c1-x65wg\" (UID: \"70844697-f6c3-4f12-8fe3-c0d80d501784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg" Nov 03 22:29:46.810506 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.810464 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvtg2\" (UniqueName: \"kubernetes.io/projected/70844697-f6c3-4f12-8fe3-c0d80d501784-kube-api-access-tvtg2\") pod \"maas-api-revoke-sub-950f43c1-x65wg\" (UID: \"70844697-f6c3-4f12-8fe3-c0d80d501784\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg" Nov 03 22:29:46.821239 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.821195 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 22:29:46.821450 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.821275 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" podUID="4324c92c-c67b-4407-868b-eecc823ac436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 22:29:46.843354 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:46.843321 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg" Nov 03 22:29:47.333887 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:47.333834 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg"] Nov 03 22:29:47.582815 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:47.582261 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 22:29:47.582815 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:47.582347 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" podUID="462d5bbd-4b7b-4c70-857a-99964a7bdef3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 22:29:48.198165 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:48.197722 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg" event={"ID":"70844697-f6c3-4f12-8fe3-c0d80d501784","Type":"ContainerStarted","Data":"69de2401c341590480f27fed4448dfbb8adf5b19a57c05bfaad71e6d8fbedf43"} Nov 03 22:29:48.198165 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:48.197823 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg" event={"ID":"70844697-f6c3-4f12-8fe3-c0d80d501784","Type":"ContainerStarted","Data":"13193048e57174f754dba7892447638fa0c6590eb65d30251b9d8aafdc969d67"} Nov 03 22:29:48.291519 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:48.291479 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:48.291981 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:48.291958 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:48.295793 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:48.295754 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:29:48.328534 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:48.328481 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg" podStartSLOduration=2.328465031 podStartE2EDuration="2.328465031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:48.32558437 +0000 UTC m=+1355.919616360" watchObservedRunningTime="2026-11-03 22:29:48.328465031 +0000 UTC m=+1355.922497005" Nov 03 22:29:48.587164 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:48.587131 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:48.587671 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:48.587658 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:48.588511 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:48.588487 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:29:49.208174 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:49.208137 5084 generic.go:356] "Generic (PLEG): container finished" podID="70844697-f6c3-4f12-8fe3-c0d80d501784" containerID="69de2401c341590480f27fed4448dfbb8adf5b19a57c05bfaad71e6d8fbedf43" exitCode=0 Nov 03 22:29:49.208600 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:49.208222 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg" event={"ID":"70844697-f6c3-4f12-8fe3-c0d80d501784","Type":"ContainerDied","Data":"69de2401c341590480f27fed4448dfbb8adf5b19a57c05bfaad71e6d8fbedf43"} Nov 03 22:29:50.387187 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:50.387150 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg" Nov 03 22:29:50.446987 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:50.446945 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70844697-f6c3-4f12-8fe3-c0d80d501784-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70844697-f6c3-4f12-8fe3-c0d80d501784-maas-api-openshift-ca\") pod \"70844697-f6c3-4f12-8fe3-c0d80d501784\" (UID: \"70844697-f6c3-4f12-8fe3-c0d80d501784\") " Nov 03 22:29:50.447248 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:50.447022 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70844697-f6c3-4f12-8fe3-c0d80d501784-kube-api-access-tvtg2\" (UniqueName: \"kubernetes.io/projected/70844697-f6c3-4f12-8fe3-c0d80d501784-kube-api-access-tvtg2\") pod \"70844697-f6c3-4f12-8fe3-c0d80d501784\" (UID: \"70844697-f6c3-4f12-8fe3-c0d80d501784\") " Nov 03 22:29:50.447248 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:50.447205 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70844697-f6c3-4f12-8fe3-c0d80d501784-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70844697-f6c3-4f12-8fe3-c0d80d501784-maas-api-xks-ca\") pod \"70844697-f6c3-4f12-8fe3-c0d80d501784\" (UID: \"70844697-f6c3-4f12-8fe3-c0d80d501784\") " Nov 03 22:29:50.447472 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:50.447443 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70844697-f6c3-4f12-8fe3-c0d80d501784-maas-api-openshift-ca" pod "70844697-f6c3-4f12-8fe3-c0d80d501784" (UID: "70844697-f6c3-4f12-8fe3-c0d80d501784"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:50.447670 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:50.447652 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70844697-f6c3-4f12-8fe3-c0d80d501784-maas-api-openshift-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:50.449294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:50.449262 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70844697-f6c3-4f12-8fe3-c0d80d501784-kube-api-access-tvtg2" pod "70844697-f6c3-4f12-8fe3-c0d80d501784" (UID: "70844697-f6c3-4f12-8fe3-c0d80d501784"). InnerVolumeSpecName "kube-api-access-tvtg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:50.449403 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:50.449293 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70844697-f6c3-4f12-8fe3-c0d80d501784-maas-api-xks-ca" pod "70844697-f6c3-4f12-8fe3-c0d80d501784" (UID: "70844697-f6c3-4f12-8fe3-c0d80d501784"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:50.549455 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:50.549419 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvtg2\" (UniqueName: \"kubernetes.io/projected/70844697-f6c3-4f12-8fe3-c0d80d501784-kube-api-access-tvtg2\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:50.549455 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:50.549454 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70844697-f6c3-4f12-8fe3-c0d80d501784-maas-api-xks-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:51.225647 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.225603 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg" event={"ID":"70844697-f6c3-4f12-8fe3-c0d80d501784","Type":"ContainerDied","Data":"13193048e57174f754dba7892447638fa0c6590eb65d30251b9d8aafdc969d67"} Nov 03 22:29:51.225647 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.225632 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg" Nov 03 22:29:51.225916 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.225657 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13193048e57174f754dba7892447638fa0c6590eb65d30251b9d8aafdc969d67" Nov 03 22:29:51.257315 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.257276 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf"] Nov 03 22:29:51.257888 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.257872 5084 cpu_manager.go:414] "containerMap: removing container" podUID="70844697-f6c3-4f12-8fe3-c0d80d501784" containerName="revoke-keys" Nov 03 22:29:51.257933 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.257889 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70844697-f6c3-4f12-8fe3-c0d80d501784" containerName="revoke-keys" Nov 03 22:29:51.257995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.257991 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="70844697-f6c3-4f12-8fe3-c0d80d501784" containerName="revoke-keys" Nov 03 22:29:51.261797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.261762 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.357459 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.357423 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf"] Nov 03 22:29:51.359735 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.359695 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.359978 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.359830 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddjhz\" (UniqueName: \"kubernetes.io/projected/8772e4ff-b94d-4424-b932-3a680dde9d73-kube-api-access-ddjhz\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.359978 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.359942 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.360141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.359978 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.360141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.360129 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.360246 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.360188 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.360296 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.360274 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8772e4ff-b94d-4424-b932-3a680dde9d73-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.360345 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.360330 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.360537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.360521 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.462294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.462248 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.462760 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.462313 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8772e4ff-b94d-4424-b932-3a680dde9d73-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.462760 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.462362 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.462760 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.462522 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.462760 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.462552 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.462760 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.462627 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddjhz\" (UniqueName: \"kubernetes.io/projected/8772e4ff-b94d-4424-b932-3a680dde9d73-kube-api-access-ddjhz\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.462760 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.462691 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.462760 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.462719 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.462760 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.462754 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.462760 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.462754 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.463222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.462820 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.463222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.463048 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8772e4ff-b94d-4424-b932-3a680dde9d73-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.463359 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.463336 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.463397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.463356 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.465406 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.465381 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.465555 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.465516 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.521748 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.521713 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddjhz\" (UniqueName: \"kubernetes.io/projected/8772e4ff-b94d-4424-b932-3a680dde9d73-kube-api-access-ddjhz\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.522558 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.522527 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.577164 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.577121 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:51.706236 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.706190 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k"] Nov 03 22:29:51.712876 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.712842 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k" Nov 03 22:29:51.755936 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.755899 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k"] Nov 03 22:29:51.766510 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.766451 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:29:51.766707 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.766562 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:29:51.767007 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.766982 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjfx7\" (UniqueName: \"kubernetes.io/projected/b6665a28-c649-487e-ab34-868ed000c53c-kube-api-access-kjfx7\") pod \"maas-api-revoke-sub-fee5eb7a-bb95k\" (UID: \"b6665a28-c649-487e-ab34-868ed000c53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k" Nov 03 22:29:51.767148 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.767022 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6665a28-c649-487e-ab34-868ed000c53c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fee5eb7a-bb95k\" (UID: \"b6665a28-c649-487e-ab34-868ed000c53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k" Nov 03 22:29:51.767278 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.767261 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6665a28-c649-487e-ab34-868ed000c53c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fee5eb7a-bb95k\" (UID: \"b6665a28-c649-487e-ab34-868ed000c53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k" Nov 03 22:29:51.870530 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.868618 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6665a28-c649-487e-ab34-868ed000c53c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fee5eb7a-bb95k\" (UID: \"b6665a28-c649-487e-ab34-868ed000c53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k" Nov 03 22:29:51.870530 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.868870 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6665a28-c649-487e-ab34-868ed000c53c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fee5eb7a-bb95k\" (UID: \"b6665a28-c649-487e-ab34-868ed000c53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k" Nov 03 22:29:51.870530 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.869000 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjfx7\" (UniqueName: \"kubernetes.io/projected/b6665a28-c649-487e-ab34-868ed000c53c-kube-api-access-kjfx7\") pod \"maas-api-revoke-sub-fee5eb7a-bb95k\" (UID: \"b6665a28-c649-487e-ab34-868ed000c53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k" Nov 03 22:29:51.870530 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.870238 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6665a28-c649-487e-ab34-868ed000c53c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fee5eb7a-bb95k\" (UID: \"b6665a28-c649-487e-ab34-868ed000c53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k" Nov 03 22:29:51.908099 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.904653 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6665a28-c649-487e-ab34-868ed000c53c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fee5eb7a-bb95k\" (UID: \"b6665a28-c649-487e-ab34-868ed000c53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k" Nov 03 22:29:51.950092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:51.946428 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjfx7\" (UniqueName: \"kubernetes.io/projected/b6665a28-c649-487e-ab34-868ed000c53c-kube-api-access-kjfx7\") pod \"maas-api-revoke-sub-fee5eb7a-bb95k\" (UID: \"b6665a28-c649-487e-ab34-868ed000c53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k" Nov 03 22:29:52.026249 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:52.026211 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf"] Nov 03 22:29:52.026549 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:29:52.026518 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8772e4ff_b94d_4424_b932_3a680dde9d73.slice/crio-c7c1583640ba15281965b7715a546eb85d5bb1ac6b22f39675010cdd6d488b74 WatchSource:0}: Error finding container c7c1583640ba15281965b7715a546eb85d5bb1ac6b22f39675010cdd6d488b74: Status 404 returned error can't find the container with id c7c1583640ba15281965b7715a546eb85d5bb1ac6b22f39675010cdd6d488b74 Nov 03 22:29:52.029135 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:52.029097 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:29:52.029234 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:52.029166 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:29:52.029234 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:52.029196 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:29:52.034866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:52.034312 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k" Nov 03 22:29:52.238730 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:52.238690 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" event={"ID":"8772e4ff-b94d-4424-b932-3a680dde9d73","Type":"ContainerStarted","Data":"f2fc48f757c2a8677e1d3fe92d9d5b3e23e7e3797696b02ae0f7304e93fff620"} Nov 03 22:29:52.238925 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:52.238750 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" event={"ID":"8772e4ff-b94d-4424-b932-3a680dde9d73","Type":"ContainerStarted","Data":"c7c1583640ba15281965b7715a546eb85d5bb1ac6b22f39675010cdd6d488b74"} Nov 03 22:29:52.363848 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:52.363789 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" podStartSLOduration=1.363769976 podStartE2EDuration="1.363769976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:52.360289945 +0000 UTC m=+1359.954321919" watchObservedRunningTime="2026-11-03 22:29:52.363769976 +0000 UTC m=+1359.957801950" Nov 03 22:29:52.423405 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:52.423298 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k"] Nov 03 22:29:52.428153 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:29:52.428115 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6665a28_c649_487e_ab34_868ed000c53c.slice/crio-6b17190d4f441eb805d8912996b8da6586471f38c47bcc09d48c8938fc27fba7 WatchSource:0}: Error finding container 6b17190d4f441eb805d8912996b8da6586471f38c47bcc09d48c8938fc27fba7: Status 404 returned error can't find the container with id 6b17190d4f441eb805d8912996b8da6586471f38c47bcc09d48c8938fc27fba7 Nov 03 22:29:52.577898 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:52.577852 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:53.248643 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:53.248601 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k" event={"ID":"b6665a28-c649-487e-ab34-868ed000c53c","Type":"ContainerStarted","Data":"908a8146a87c0a0ea0543e29b02f8ccfa7f6c3565c09785acbf1e83b1f909589"} Nov 03 22:29:53.248869 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:53.248663 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k" event={"ID":"b6665a28-c649-487e-ab34-868ed000c53c","Type":"ContainerStarted","Data":"6b17190d4f441eb805d8912996b8da6586471f38c47bcc09d48c8938fc27fba7"} Nov 03 22:29:53.386004 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:53.385926 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k" podStartSLOduration=2.385899697 podStartE2EDuration="2.385899697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:53.381240333 +0000 UTC m=+1360.975272307" watchObservedRunningTime="2026-11-03 22:29:53.385899697 +0000 UTC m=+1360.979931702" Nov 03 22:29:53.580086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:53.579625 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:29:53.580086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:53.579692 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" podUID="8772e4ff-b94d-4424-b932-3a680dde9d73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 22:29:54.282425 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:54.282387 5084 generic.go:356] "Generic (PLEG): container finished" podID="b6665a28-c649-487e-ab34-868ed000c53c" containerID="908a8146a87c0a0ea0543e29b02f8ccfa7f6c3565c09785acbf1e83b1f909589" exitCode=0 Nov 03 22:29:54.282597 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:54.282486 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k" event={"ID":"b6665a28-c649-487e-ab34-868ed000c53c","Type":"ContainerDied","Data":"908a8146a87c0a0ea0543e29b02f8ccfa7f6c3565c09785acbf1e83b1f909589"} Nov 03 22:29:54.424374 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:54.424327 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7091-155d03"] Nov 03 22:29:54.428669 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:54.428639 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-155d03" Nov 03 22:29:54.484695 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:54.484657 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jj8tk\"" Nov 03 22:29:54.485079 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:54.485032 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7091-155d03"] Nov 03 22:29:54.510559 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:54.510518 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c8qg\" (UniqueName: \"kubernetes.io/projected/6157bbfe-8735-42e5-b0c5-131f4315cf87-kube-api-access-6c8qg\") pod \"test-curl-7091-155d03\" (UID: \"6157bbfe-8735-42e5-b0c5-131f4315cf87\") " pod="opendatahub/test-curl-7091-155d03" Nov 03 22:29:54.581159 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:54.580505 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:29:54.581159 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:54.580580 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" podUID="8772e4ff-b94d-4424-b932-3a680dde9d73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 22:29:54.612523 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:54.612481 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8qg\" (UniqueName: \"kubernetes.io/projected/6157bbfe-8735-42e5-b0c5-131f4315cf87-kube-api-access-6c8qg\") pod \"test-curl-7091-155d03\" (UID: \"6157bbfe-8735-42e5-b0c5-131f4315cf87\") " pod="opendatahub/test-curl-7091-155d03" Nov 03 22:29:54.682268 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:54.682218 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c8qg\" (UniqueName: \"kubernetes.io/projected/6157bbfe-8735-42e5-b0c5-131f4315cf87-kube-api-access-6c8qg\") pod \"test-curl-7091-155d03\" (UID: \"6157bbfe-8735-42e5-b0c5-131f4315cf87\") " pod="opendatahub/test-curl-7091-155d03" Nov 03 22:29:54.741884 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:54.741836 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-155d03" Nov 03 22:29:55.189019 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:29:55.188936 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6157bbfe_8735_42e5_b0c5_131f4315cf87.slice/crio-166953e55b69fc051254fce1f77150be0cc66fe6e37e54ec6798c085df300dfc WatchSource:0}: Error finding container 166953e55b69fc051254fce1f77150be0cc66fe6e37e54ec6798c085df300dfc: Status 404 returned error can't find the container with id 166953e55b69fc051254fce1f77150be0cc66fe6e37e54ec6798c085df300dfc Nov 03 22:29:55.191211 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:55.191185 5084 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:29:55.192447 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:55.192423 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7091-155d03"] Nov 03 22:29:55.293158 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:55.293122 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7091-155d03" event={"ID":"6157bbfe-8735-42e5-b0c5-131f4315cf87","Type":"ContainerStarted","Data":"166953e55b69fc051254fce1f77150be0cc66fe6e37e54ec6798c085df300dfc"} Nov 03 22:29:55.498852 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:55.498815 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k" Nov 03 22:29:55.524913 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:55.524882 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6665a28-c649-487e-ab34-868ed000c53c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6665a28-c649-487e-ab34-868ed000c53c-maas-api-openshift-ca\") pod \"b6665a28-c649-487e-ab34-868ed000c53c\" (UID: \"b6665a28-c649-487e-ab34-868ed000c53c\") " Nov 03 22:29:55.525141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:55.524961 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6665a28-c649-487e-ab34-868ed000c53c-kube-api-access-kjfx7\" (UniqueName: \"kubernetes.io/projected/b6665a28-c649-487e-ab34-868ed000c53c-kube-api-access-kjfx7\") pod \"b6665a28-c649-487e-ab34-868ed000c53c\" (UID: \"b6665a28-c649-487e-ab34-868ed000c53c\") " Nov 03 22:29:55.525233 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:55.525217 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6665a28-c649-487e-ab34-868ed000c53c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6665a28-c649-487e-ab34-868ed000c53c-maas-api-xks-ca\") pod \"b6665a28-c649-487e-ab34-868ed000c53c\" (UID: \"b6665a28-c649-487e-ab34-868ed000c53c\") " Nov 03 22:29:55.525322 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:55.525291 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6665a28-c649-487e-ab34-868ed000c53c-maas-api-openshift-ca" pod "b6665a28-c649-487e-ab34-868ed000c53c" (UID: "b6665a28-c649-487e-ab34-868ed000c53c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:55.525670 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:55.525653 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6665a28-c649-487e-ab34-868ed000c53c-maas-api-openshift-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:55.527448 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:55.527421 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6665a28-c649-487e-ab34-868ed000c53c-kube-api-access-kjfx7" pod "b6665a28-c649-487e-ab34-868ed000c53c" (UID: "b6665a28-c649-487e-ab34-868ed000c53c"). InnerVolumeSpecName "kube-api-access-kjfx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:55.527527 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:55.527462 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6665a28-c649-487e-ab34-868ed000c53c-maas-api-xks-ca" pod "b6665a28-c649-487e-ab34-868ed000c53c" (UID: "b6665a28-c649-487e-ab34-868ed000c53c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:55.580264 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:55.580216 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:29:55.580463 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:55.580302 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" podUID="8772e4ff-b94d-4424-b932-3a680dde9d73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 22:29:55.627397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:55.627364 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjfx7\" (UniqueName: \"kubernetes.io/projected/b6665a28-c649-487e-ab34-868ed000c53c-kube-api-access-kjfx7\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:55.627397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:55.627394 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6665a28-c649-487e-ab34-868ed000c53c-maas-api-xks-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:29:56.311789 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:56.311756 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k" event={"ID":"b6665a28-c649-487e-ab34-868ed000c53c","Type":"ContainerDied","Data":"6b17190d4f441eb805d8912996b8da6586471f38c47bcc09d48c8938fc27fba7"} Nov 03 22:29:56.311964 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:56.311808 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b17190d4f441eb805d8912996b8da6586471f38c47bcc09d48c8938fc27fba7" Nov 03 22:29:56.311964 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:56.311863 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k" Nov 03 22:29:56.579169 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:56.578464 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:29:56.579169 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:56.578525 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" podUID="8772e4ff-b94d-4424-b932-3a680dde9d73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 22:29:57.321600 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:57.321559 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7091-155d03" event={"ID":"6157bbfe-8735-42e5-b0c5-131f4315cf87","Type":"ContainerStarted","Data":"2fdb9e03aee703c7ca6b8e4e04f326326c7e7da4dc9e30d608b9f84e0030c359"} Nov 03 22:29:57.446224 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:57.446160 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7091-155d03" podStartSLOduration=2.211261992 podStartE2EDuration="3.446144084s" totalImagesPullingTime="1.234882092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:57.443473243 +0000 UTC m=+1365.037505218" watchObservedRunningTime="2026-11-03 22:29:57.446144084 +0000 UTC m=+1365.040176058" Nov 03 22:29:57.580595 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:57.580496 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:29:57.580595 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:57.580565 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" podUID="8772e4ff-b94d-4424-b932-3a680dde9d73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 22:29:58.578229 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:58.578181 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 22:29:58.578724 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:58.578256 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" podUID="8772e4ff-b94d-4424-b932-3a680dde9d73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 22:29:58.959130 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:58.959014 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7091-155d03"] Nov 03 22:29:58.963128 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:58.963081 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7091-155d03"] Nov 03 22:29:59.125690 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:59.125634 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7091-86603f"] Nov 03 22:29:59.126655 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:59.126625 5084 cpu_manager.go:414] "containerMap: removing container" podUID="6157bbfe-8735-42e5-b0c5-131f4315cf87" containerName="test-curl-7091-155d03" Nov 03 22:29:59.126795 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:59.126658 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6157bbfe-8735-42e5-b0c5-131f4315cf87" containerName="test-curl-7091-155d03" Nov 03 22:29:59.126795 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:59.126696 5084 cpu_manager.go:414] "containerMap: removing container" podUID="b6665a28-c649-487e-ab34-868ed000c53c" containerName="revoke-keys" Nov 03 22:29:59.126795 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:59.126704 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6665a28-c649-487e-ab34-868ed000c53c" containerName="revoke-keys" Nov 03 22:29:59.126967 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:59.126952 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="6157bbfe-8735-42e5-b0c5-131f4315cf87" containerName="test-curl-7091-155d03" Nov 03 22:29:59.127019 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:59.126970 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6665a28-c649-487e-ab34-868ed000c53c" containerName="revoke-keys" Nov 03 22:29:59.131581 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:59.131543 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-86603f" Nov 03 22:29:59.180397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:59.180352 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7091-86603f"] Nov 03 22:29:59.213472 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:59.213349 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvcj4\" (UniqueName: \"kubernetes.io/projected/5b262f86-3e06-4e2e-ac5f-95872e882179-kube-api-access-cvcj4\") pod \"test-curl-7091-86603f\" (UID: \"5b262f86-3e06-4e2e-ac5f-95872e882179\") " pod="opendatahub/test-curl-7091-86603f" Nov 03 22:29:59.315540 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:59.315491 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvcj4\" (UniqueName: \"kubernetes.io/projected/5b262f86-3e06-4e2e-ac5f-95872e882179-kube-api-access-cvcj4\") pod \"test-curl-7091-86603f\" (UID: \"5b262f86-3e06-4e2e-ac5f-95872e882179\") " pod="opendatahub/test-curl-7091-86603f" Nov 03 22:29:59.340180 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:59.340106 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7091-155d03" podUID="6157bbfe-8735-42e5-b0c5-131f4315cf87" containerName="test-curl-7091-155d03" containerID="cri-o://2fdb9e03aee703c7ca6b8e4e04f326326c7e7da4dc9e30d608b9f84e0030c359" gracePeriod=2 Nov 03 22:29:59.376209 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:59.376122 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvcj4\" (UniqueName: \"kubernetes.io/projected/5b262f86-3e06-4e2e-ac5f-95872e882179-kube-api-access-cvcj4\") pod \"test-curl-7091-86603f\" (UID: \"5b262f86-3e06-4e2e-ac5f-95872e882179\") " pod="opendatahub/test-curl-7091-86603f" Nov 03 22:29:59.392894 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:29:59.392758 5084 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-155d03\" is forbidden: User \"system:node:ip-10-0-141-84.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-84.us-west-2.compute.internal' and this object" podUID="6157bbfe-8735-42e5-b0c5-131f4315cf87" pod="opendatahub/test-curl-7091-155d03" Nov 03 22:29:59.449328 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:59.449283 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-86603f" Nov 03 22:29:59.583723 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:59.582426 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:59.583723 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:59.583570 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:59.584526 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:59.584502 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:29:59.758095 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:29:59.757990 5084 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-155d03\" is forbidden: User \"system:node:ip-10-0-141-84.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-84.us-west-2.compute.internal' and this object" podUID="6157bbfe-8735-42e5-b0c5-131f4315cf87" pod="opendatahub/test-curl-7091-155d03" Nov 03 22:29:59.810644 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:29:59.810585 5084 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-155d03\" is forbidden: User \"system:node:ip-10-0-141-84.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-84.us-west-2.compute.internal' and this object" podUID="6157bbfe-8735-42e5-b0c5-131f4315cf87" pod="opendatahub/test-curl-7091-155d03" Nov 03 22:29:59.850512 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:29:59.850478 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7091-86603f"] Nov 03 22:29:59.851157 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:29:59.851129 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b262f86_3e06_4e2e_ac5f_95872e882179.slice/crio-eb63439d79c61c7991e38d62e2231736b45a0c8905bd45cc23eb9f09f674f55b WatchSource:0}: Error finding container eb63439d79c61c7991e38d62e2231736b45a0c8905bd45cc23eb9f09f674f55b: Status 404 returned error can't find the container with id eb63439d79c61c7991e38d62e2231736b45a0c8905bd45cc23eb9f09f674f55b Nov 03 22:30:00.212549 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.212511 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k"] Nov 03 22:30:00.217075 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.217032 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" Nov 03 22:30:00.225575 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.225322 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g"] Nov 03 22:30:00.230636 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.230608 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g" Nov 03 22:30:00.233596 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.233561 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st"] Nov 03 22:30:00.237938 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.237912 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" Nov 03 22:30:00.257031 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.256995 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd"] Nov 03 22:30:00.261762 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.261735 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd" Nov 03 22:30:00.271791 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.270983 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:30:00.274864 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.274651 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k"] Nov 03 22:30:00.294217 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.294179 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g"] Nov 03 22:30:00.306328 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.306286 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st"] Nov 03 22:30:00.342859 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.342822 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66bp4\" (UniqueName: \"kubernetes.io/projected/70260099-baab-410c-9f3f-9ab097b74a86-kube-api-access-66bp4\") pod \"maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g\" (UID: \"70260099-baab-410c-9f3f-9ab097b74a86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g" Nov 03 22:30:00.343076 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.342879 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70260099-baab-410c-9f3f-9ab097b74a86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g\" (UID: \"70260099-baab-410c-9f3f-9ab097b74a86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g" Nov 03 22:30:00.343076 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.343006 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6r7t\" (UniqueName: \"kubernetes.io/projected/d7bf5b25-5684-439a-980c-bcb503e073b4-kube-api-access-v6r7t\") pod \"maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st\" (UID: \"d7bf5b25-5684-439a-980c-bcb503e073b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" Nov 03 22:30:00.343229 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.343128 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qzc8\" (UniqueName: \"kubernetes.io/projected/75d60c1c-e3ab-4514-9f51-91f56de520a9-kube-api-access-5qzc8\") pod \"maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k\" (UID: \"75d60c1c-e3ab-4514-9f51-91f56de520a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" Nov 03 22:30:00.343292 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.343229 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d60c1c-e3ab-4514-9f51-91f56de520a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k\" (UID: \"75d60c1c-e3ab-4514-9f51-91f56de520a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" Nov 03 22:30:00.343375 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.343359 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bf5b25-5684-439a-980c-bcb503e073b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st\" (UID: \"d7bf5b25-5684-439a-980c-bcb503e073b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" Nov 03 22:30:00.355226 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.355181 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7091-86603f" event={"ID":"5b262f86-3e06-4e2e-ac5f-95872e882179","Type":"ContainerStarted","Data":"458c279ec72d2d781e304f2b13b19ddb2a1c2598ad3449f19bca1199c6889a42"} Nov 03 22:30:00.355402 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.355247 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7091-86603f" event={"ID":"5b262f86-3e06-4e2e-ac5f-95872e882179","Type":"ContainerStarted","Data":"eb63439d79c61c7991e38d62e2231736b45a0c8905bd45cc23eb9f09f674f55b"} Nov 03 22:30:00.356205 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.356176 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd"] Nov 03 22:30:00.385649 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:00.385603 5084 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-155d03\" is forbidden: User \"system:node:ip-10-0-141-84.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-84.us-west-2.compute.internal' and this object" podUID="6157bbfe-8735-42e5-b0c5-131f4315cf87" pod="opendatahub/test-curl-7091-155d03" Nov 03 22:30:00.444410 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.444367 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6r7t\" (UniqueName: \"kubernetes.io/projected/d7bf5b25-5684-439a-980c-bcb503e073b4-kube-api-access-v6r7t\") pod \"maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st\" (UID: \"d7bf5b25-5684-439a-980c-bcb503e073b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" Nov 03 22:30:00.444410 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.444414 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrclw\" (UniqueName: \"kubernetes.io/projected/3f65be02-e914-4d38-917e-5221c7ab028b-kube-api-access-wrclw\") pod \"maas-api-key-cleanup-29895750-gqfsd\" (UID: \"3f65be02-e914-4d38-917e-5221c7ab028b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd" Nov 03 22:30:00.444614 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.444471 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qzc8\" (UniqueName: \"kubernetes.io/projected/75d60c1c-e3ab-4514-9f51-91f56de520a9-kube-api-access-5qzc8\") pod \"maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k\" (UID: \"75d60c1c-e3ab-4514-9f51-91f56de520a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" Nov 03 22:30:00.444614 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.444591 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f65be02-e914-4d38-917e-5221c7ab028b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895750-gqfsd\" (UID: \"3f65be02-e914-4d38-917e-5221c7ab028b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd" Nov 03 22:30:00.444763 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.444734 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d60c1c-e3ab-4514-9f51-91f56de520a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k\" (UID: \"75d60c1c-e3ab-4514-9f51-91f56de520a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" Nov 03 22:30:00.444857 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.444837 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bf5b25-5684-439a-980c-bcb503e073b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st\" (UID: \"d7bf5b25-5684-439a-980c-bcb503e073b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" Nov 03 22:30:00.444966 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.444944 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66bp4\" (UniqueName: \"kubernetes.io/projected/70260099-baab-410c-9f3f-9ab097b74a86-kube-api-access-66bp4\") pod \"maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g\" (UID: \"70260099-baab-410c-9f3f-9ab097b74a86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g" Nov 03 22:30:00.445183 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.445163 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70260099-baab-410c-9f3f-9ab097b74a86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g\" (UID: \"70260099-baab-410c-9f3f-9ab097b74a86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g" Nov 03 22:30:00.445687 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.445664 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bf5b25-5684-439a-980c-bcb503e073b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st\" (UID: \"d7bf5b25-5684-439a-980c-bcb503e073b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" Nov 03 22:30:00.445756 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.445695 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d60c1c-e3ab-4514-9f51-91f56de520a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k\" (UID: \"75d60c1c-e3ab-4514-9f51-91f56de520a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" Nov 03 22:30:00.445873 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.445856 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70260099-baab-410c-9f3f-9ab097b74a86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g\" (UID: \"70260099-baab-410c-9f3f-9ab097b74a86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g" Nov 03 22:30:00.503796 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.503723 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6r7t\" (UniqueName: \"kubernetes.io/projected/d7bf5b25-5684-439a-980c-bcb503e073b4-kube-api-access-v6r7t\") pod \"maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st\" (UID: \"d7bf5b25-5684-439a-980c-bcb503e073b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" Nov 03 22:30:00.506616 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.506582 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qzc8\" (UniqueName: \"kubernetes.io/projected/75d60c1c-e3ab-4514-9f51-91f56de520a9-kube-api-access-5qzc8\") pod \"maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k\" (UID: \"75d60c1c-e3ab-4514-9f51-91f56de520a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" Nov 03 22:30:00.506750 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.506639 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66bp4\" (UniqueName: \"kubernetes.io/projected/70260099-baab-410c-9f3f-9ab097b74a86-kube-api-access-66bp4\") pod \"maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g\" (UID: \"70260099-baab-410c-9f3f-9ab097b74a86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g" Nov 03 22:30:00.509229 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.509178 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7091-86603f" podStartSLOduration=1.509162204 podStartE2EDuration="1.509162204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:00.50275203 +0000 UTC m=+1368.096784005" watchObservedRunningTime="2026-11-03 22:30:00.509162204 +0000 UTC m=+1368.103194177" Nov 03 22:30:00.533701 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.533663 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" Nov 03 22:30:00.546830 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.546791 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f65be02-e914-4d38-917e-5221c7ab028b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895750-gqfsd\" (UID: \"3f65be02-e914-4d38-917e-5221c7ab028b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd" Nov 03 22:30:00.547001 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.546915 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g" Nov 03 22:30:00.547001 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.546957 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrclw\" (UniqueName: \"kubernetes.io/projected/3f65be02-e914-4d38-917e-5221c7ab028b-kube-api-access-wrclw\") pod \"maas-api-key-cleanup-29895750-gqfsd\" (UID: \"3f65be02-e914-4d38-917e-5221c7ab028b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd" Nov 03 22:30:00.547497 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.547469 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f65be02-e914-4d38-917e-5221c7ab028b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895750-gqfsd\" (UID: \"3f65be02-e914-4d38-917e-5221c7ab028b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd" Nov 03 22:30:00.555160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.555124 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" Nov 03 22:30:00.606762 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.606531 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrclw\" (UniqueName: \"kubernetes.io/projected/3f65be02-e914-4d38-917e-5221c7ab028b-kube-api-access-wrclw\") pod \"maas-api-key-cleanup-29895750-gqfsd\" (UID: \"3f65be02-e914-4d38-917e-5221c7ab028b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd" Nov 03 22:30:00.884937 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.884896 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd" Nov 03 22:30:00.928444 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:00.928403 5084 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-155d03\" is forbidden: User \"system:node:ip-10-0-141-84.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-84.us-west-2.compute.internal' and this object" podUID="6157bbfe-8735-42e5-b0c5-131f4315cf87" pod="opendatahub/test-curl-7091-155d03" Nov 03 22:30:00.939825 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.939784 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k"] Nov 03 22:30:00.941878 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:30:00.941832 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d60c1c_e3ab_4514_9f51_91f56de520a9.slice/crio-9683766be29b362b2cada01748c6a3ec0f5a55da355b727dd444bbb48d60d502 WatchSource:0}: Error finding container 9683766be29b362b2cada01748c6a3ec0f5a55da355b727dd444bbb48d60d502: Status 404 returned error can't find the container with id 9683766be29b362b2cada01748c6a3ec0f5a55da355b727dd444bbb48d60d502 Nov 03 22:30:00.991152 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:30:00.991118 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7bf5b25_5684_439a_980c_bcb503e073b4.slice/crio-579393fdbb382736a8209ff7d704c486e0c9d022478e8188a29d8800b4973532 WatchSource:0}: Error finding container 579393fdbb382736a8209ff7d704c486e0c9d022478e8188a29d8800b4973532: Status 404 returned error can't find the container with id 579393fdbb382736a8209ff7d704c486e0c9d022478e8188a29d8800b4973532 Nov 03 22:30:00.992603 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:30:00.992572 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70260099_baab_410c_9f3f_9ab097b74a86.slice/crio-d3613faed3e3eaeb49aa0e535835565eaedbe7d4868028545d371d30a9fff6d3 WatchSource:0}: Error finding container d3613faed3e3eaeb49aa0e535835565eaedbe7d4868028545d371d30a9fff6d3: Status 404 returned error can't find the container with id d3613faed3e3eaeb49aa0e535835565eaedbe7d4868028545d371d30a9fff6d3 Nov 03 22:30:00.993009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.992988 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st"] Nov 03 22:30:00.997865 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:00.997830 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g"] Nov 03 22:30:01.324027 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:01.323992 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd"] Nov 03 22:30:01.324257 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:30:01.324179 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f65be02_e914_4d38_917e_5221c7ab028b.slice/crio-869ece2b24e7ea1231e99443f6561b0965895650b565df650632d07fc969c4a8 WatchSource:0}: Error finding container 869ece2b24e7ea1231e99443f6561b0965895650b565df650632d07fc969c4a8: Status 404 returned error can't find the container with id 869ece2b24e7ea1231e99443f6561b0965895650b565df650632d07fc969c4a8 Nov 03 22:30:01.366755 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:01.366722 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" event={"ID":"75d60c1c-e3ab-4514-9f51-91f56de520a9","Type":"ContainerStarted","Data":"9683766be29b362b2cada01748c6a3ec0f5a55da355b727dd444bbb48d60d502"} Nov 03 22:30:01.368646 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:01.368612 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" event={"ID":"d7bf5b25-5684-439a-980c-bcb503e073b4","Type":"ContainerStarted","Data":"579393fdbb382736a8209ff7d704c486e0c9d022478e8188a29d8800b4973532"} Nov 03 22:30:01.370271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:01.370236 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd" event={"ID":"3f65be02-e914-4d38-917e-5221c7ab028b","Type":"ContainerStarted","Data":"869ece2b24e7ea1231e99443f6561b0965895650b565df650632d07fc969c4a8"} Nov 03 22:30:01.371697 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:01.371662 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g" event={"ID":"70260099-baab-410c-9f3f-9ab097b74a86","Type":"ContainerStarted","Data":"d3613faed3e3eaeb49aa0e535835565eaedbe7d4868028545d371d30a9fff6d3"} Nov 03 22:30:01.622624 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:01.622595 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-155d03" Nov 03 22:30:01.772715 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:01.772671 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6157bbfe-8735-42e5-b0c5-131f4315cf87-kube-api-access-6c8qg\" (UniqueName: \"kubernetes.io/projected/6157bbfe-8735-42e5-b0c5-131f4315cf87-kube-api-access-6c8qg\") pod \"6157bbfe-8735-42e5-b0c5-131f4315cf87\" (UID: \"6157bbfe-8735-42e5-b0c5-131f4315cf87\") " Nov 03 22:30:01.775104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:01.775036 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6157bbfe-8735-42e5-b0c5-131f4315cf87-kube-api-access-6c8qg" pod "6157bbfe-8735-42e5-b0c5-131f4315cf87" (UID: "6157bbfe-8735-42e5-b0c5-131f4315cf87"). InnerVolumeSpecName "kube-api-access-6c8qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:01.874509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:01.874471 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c8qg\" (UniqueName: \"kubernetes.io/projected/6157bbfe-8735-42e5-b0c5-131f4315cf87-kube-api-access-6c8qg\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:02.153735 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.153692 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7091-86603f"] Nov 03 22:30:02.159673 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.159616 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7091-86603f"] Nov 03 22:30:02.383716 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.383681 5084 generic.go:356] "Generic (PLEG): container finished" podID="75d60c1c-e3ab-4514-9f51-91f56de520a9" containerID="1143b19ceeb6b62d1cedd27f187767040cb0b8784abd0bf82aa2fa4362a72e4c" exitCode=60 Nov 03 22:30:02.383903 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.383759 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" event={"ID":"75d60c1c-e3ab-4514-9f51-91f56de520a9","Type":"ContainerDied","Data":"1143b19ceeb6b62d1cedd27f187767040cb0b8784abd0bf82aa2fa4362a72e4c"} Nov 03 22:30:02.384160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.384046 5084 scope.go:122] "RemoveContainer" containerID="1143b19ceeb6b62d1cedd27f187767040cb0b8784abd0bf82aa2fa4362a72e4c" Nov 03 22:30:02.386197 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.386168 5084 generic.go:356] "Generic (PLEG): container finished" podID="d7bf5b25-5684-439a-980c-bcb503e073b4" containerID="295fc11fa47723806095ceea9bba88e74d313d56316497653a92a013bf985a99" exitCode=60 Nov 03 22:30:02.386336 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.386246 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" event={"ID":"d7bf5b25-5684-439a-980c-bcb503e073b4","Type":"ContainerDied","Data":"295fc11fa47723806095ceea9bba88e74d313d56316497653a92a013bf985a99"} Nov 03 22:30:02.386537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.386518 5084 scope.go:122] "RemoveContainer" containerID="295fc11fa47723806095ceea9bba88e74d313d56316497653a92a013bf985a99" Nov 03 22:30:02.388403 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.388299 5084 generic.go:356] "Generic (PLEG): container finished" podID="3f65be02-e914-4d38-917e-5221c7ab028b" containerID="58530b651bd92fa0908373ab65ca71aa7a7d7bf668b4437df30833c404c96046" exitCode=60 Nov 03 22:30:02.388403 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.388337 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd" event={"ID":"3f65be02-e914-4d38-917e-5221c7ab028b","Type":"ContainerDied","Data":"58530b651bd92fa0908373ab65ca71aa7a7d7bf668b4437df30833c404c96046"} Nov 03 22:30:02.388659 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.388638 5084 scope.go:122] "RemoveContainer" containerID="58530b651bd92fa0908373ab65ca71aa7a7d7bf668b4437df30833c404c96046" Nov 03 22:30:02.390237 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.390213 5084 generic.go:356] "Generic (PLEG): container finished" podID="6157bbfe-8735-42e5-b0c5-131f4315cf87" containerID="2fdb9e03aee703c7ca6b8e4e04f326326c7e7da4dc9e30d608b9f84e0030c359" exitCode=137 Nov 03 22:30:02.390362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.390266 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-155d03" Nov 03 22:30:02.390362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.390305 5084 scope.go:122] "RemoveContainer" containerID="2fdb9e03aee703c7ca6b8e4e04f326326c7e7da4dc9e30d608b9f84e0030c359" Nov 03 22:30:02.392315 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.392285 5084 generic.go:356] "Generic (PLEG): container finished" podID="70260099-baab-410c-9f3f-9ab097b74a86" containerID="6e75af9551dc20136acd6749b48f333754bd187286c6bff3332501828d292d0f" exitCode=60 Nov 03 22:30:02.392477 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.392410 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g" event={"ID":"70260099-baab-410c-9f3f-9ab097b74a86","Type":"ContainerDied","Data":"6e75af9551dc20136acd6749b48f333754bd187286c6bff3332501828d292d0f"} Nov 03 22:30:02.392590 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.392558 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7091-86603f" podUID="5b262f86-3e06-4e2e-ac5f-95872e882179" containerName="test-curl-7091-86603f" containerID="cri-o://458c279ec72d2d781e304f2b13b19ddb2a1c2598ad3449f19bca1199c6889a42" gracePeriod=2 Nov 03 22:30:02.393334 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.393315 5084 scope.go:122] "RemoveContainer" containerID="6e75af9551dc20136acd6749b48f333754bd187286c6bff3332501828d292d0f" Nov 03 22:30:02.405509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.405477 5084 scope.go:122] "RemoveContainer" containerID="2fdb9e03aee703c7ca6b8e4e04f326326c7e7da4dc9e30d608b9f84e0030c359" Nov 03 22:30:02.406126 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:02.406091 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fdb9e03aee703c7ca6b8e4e04f326326c7e7da4dc9e30d608b9f84e0030c359\": container with ID starting with 2fdb9e03aee703c7ca6b8e4e04f326326c7e7da4dc9e30d608b9f84e0030c359 not found: ID does not exist" containerID="2fdb9e03aee703c7ca6b8e4e04f326326c7e7da4dc9e30d608b9f84e0030c359" Nov 03 22:30:02.406259 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.406152 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fdb9e03aee703c7ca6b8e4e04f326326c7e7da4dc9e30d608b9f84e0030c359"} err="failed to get container status \"2fdb9e03aee703c7ca6b8e4e04f326326c7e7da4dc9e30d608b9f84e0030c359\": rpc error: code = NotFound desc = could not find container \"2fdb9e03aee703c7ca6b8e4e04f326326c7e7da4dc9e30d608b9f84e0030c359\": container with ID starting with 2fdb9e03aee703c7ca6b8e4e04f326326c7e7da4dc9e30d608b9f84e0030c359 not found: ID does not exist" Nov 03 22:30:02.553794 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:02.553733 5084 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-155d03\" is forbidden: User \"system:node:ip-10-0-141-84.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-84.us-west-2.compute.internal' and this object" podUID="6157bbfe-8735-42e5-b0c5-131f4315cf87" pod="opendatahub/test-curl-7091-155d03" Nov 03 22:30:02.606332 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:02.606286 5084 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-86603f\" is forbidden: User \"system:node:ip-10-0-141-84.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-84.us-west-2.compute.internal' and this object" podUID="5b262f86-3e06-4e2e-ac5f-95872e882179" pod="opendatahub/test-curl-7091-86603f" Nov 03 22:30:02.630570 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:02.630487 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6157bbfe-8735-42e5-b0c5-131f4315cf87" path="/var/lib/kubelet/pods/6157bbfe-8735-42e5-b0c5-131f4315cf87/volumes" Nov 03 22:30:03.405328 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:03.405284 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" event={"ID":"75d60c1c-e3ab-4514-9f51-91f56de520a9","Type":"ContainerStarted","Data":"d4e0f46f9835c3e5df4bfd3d8c95f3668a05275879fd028b86d57d256bd0888d"} Nov 03 22:30:03.407671 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:03.407625 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" event={"ID":"d7bf5b25-5684-439a-980c-bcb503e073b4","Type":"ContainerStarted","Data":"8f3d3904b8cf268aa9ada1f73413c8a9f478242b0e7e89573cc1dea213f07b4b"} Nov 03 22:30:03.409995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:03.409961 5084 generic.go:356] "Generic (PLEG): container finished" podID="3f65be02-e914-4d38-917e-5221c7ab028b" containerID="346e62c91ab2f8b54faba4ef2f1ab736cd95d9964b4a0bdd26c1b28982522cc5" exitCode=60 Nov 03 22:30:03.410281 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:03.410045 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd" event={"ID":"3f65be02-e914-4d38-917e-5221c7ab028b","Type":"ContainerDied","Data":"346e62c91ab2f8b54faba4ef2f1ab736cd95d9964b4a0bdd26c1b28982522cc5"} Nov 03 22:30:03.410281 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:03.410121 5084 scope.go:122] "RemoveContainer" containerID="58530b651bd92fa0908373ab65ca71aa7a7d7bf668b4437df30833c404c96046" Nov 03 22:30:03.410476 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:03.410454 5084 scope.go:122] "RemoveContainer" containerID="346e62c91ab2f8b54faba4ef2f1ab736cd95d9964b4a0bdd26c1b28982522cc5" Nov 03 22:30:03.410846 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:03.410816 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895750-gqfsd_odh-ai-gateway-infra(3f65be02-e914-4d38-917e-5221c7ab028b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd" podUID="3f65be02-e914-4d38-917e-5221c7ab028b" Nov 03 22:30:03.413702 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:03.413661 5084 generic.go:356] "Generic (PLEG): container finished" podID="70260099-baab-410c-9f3f-9ab097b74a86" containerID="43c73bf81497204d1ba90fe90f87d4b026e01968505fa302ebd47e37d48da557" exitCode=60 Nov 03 22:30:03.413846 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:03.413735 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g" event={"ID":"70260099-baab-410c-9f3f-9ab097b74a86","Type":"ContainerDied","Data":"43c73bf81497204d1ba90fe90f87d4b026e01968505fa302ebd47e37d48da557"} Nov 03 22:30:03.414137 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:03.414112 5084 scope.go:122] "RemoveContainer" containerID="43c73bf81497204d1ba90fe90f87d4b026e01968505fa302ebd47e37d48da557" Nov 03 22:30:03.414511 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:03.414476 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g_odh-ai-gateway-infra(70260099-baab-410c-9f3f-9ab097b74a86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g" podUID="70260099-baab-410c-9f3f-9ab097b74a86" Nov 03 22:30:03.527197 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:03.527139 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" podStartSLOduration=3.527122126 podStartE2EDuration="3.527122126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:03.522419736 +0000 UTC m=+1371.116451710" watchObservedRunningTime="2026-11-03 22:30:03.527122126 +0000 UTC m=+1371.121154099" Nov 03 22:30:03.606018 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:03.605990 5084 scope.go:122] "RemoveContainer" containerID="6e75af9551dc20136acd6749b48f333754bd187286c6bff3332501828d292d0f" Nov 03 22:30:03.642043 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:03.641976 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" podStartSLOduration=3.641958267 podStartE2EDuration="3.641958267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:03.638808873 +0000 UTC m=+1371.232840847" watchObservedRunningTime="2026-11-03 22:30:03.641958267 +0000 UTC m=+1371.235990240" Nov 03 22:30:04.424436 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:04.424397 5084 generic.go:356] "Generic (PLEG): container finished" podID="d7bf5b25-5684-439a-980c-bcb503e073b4" containerID="8f3d3904b8cf268aa9ada1f73413c8a9f478242b0e7e89573cc1dea213f07b4b" exitCode=60 Nov 03 22:30:04.424643 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:04.424497 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" event={"ID":"d7bf5b25-5684-439a-980c-bcb503e073b4","Type":"ContainerDied","Data":"8f3d3904b8cf268aa9ada1f73413c8a9f478242b0e7e89573cc1dea213f07b4b"} Nov 03 22:30:04.424643 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:04.424600 5084 scope.go:122] "RemoveContainer" containerID="295fc11fa47723806095ceea9bba88e74d313d56316497653a92a013bf985a99" Nov 03 22:30:04.424839 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:04.424816 5084 scope.go:122] "RemoveContainer" containerID="8f3d3904b8cf268aa9ada1f73413c8a9f478242b0e7e89573cc1dea213f07b4b" Nov 03 22:30:04.425262 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:04.425222 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st_odh-ai-gateway-infra(d7bf5b25-5684-439a-980c-bcb503e073b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" podUID="d7bf5b25-5684-439a-980c-bcb503e073b4" Nov 03 22:30:04.427403 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:04.427373 5084 scope.go:122] "RemoveContainer" containerID="346e62c91ab2f8b54faba4ef2f1ab736cd95d9964b4a0bdd26c1b28982522cc5" Nov 03 22:30:04.427743 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:04.427716 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895750-gqfsd_odh-ai-gateway-infra(3f65be02-e914-4d38-917e-5221c7ab028b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd" podUID="3f65be02-e914-4d38-917e-5221c7ab028b" Nov 03 22:30:04.429167 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:04.429149 5084 scope.go:122] "RemoveContainer" containerID="43c73bf81497204d1ba90fe90f87d4b026e01968505fa302ebd47e37d48da557" Nov 03 22:30:04.429470 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:04.429442 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g_odh-ai-gateway-infra(70260099-baab-410c-9f3f-9ab097b74a86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g" podUID="70260099-baab-410c-9f3f-9ab097b74a86" Nov 03 22:30:04.431081 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:04.431043 5084 generic.go:356] "Generic (PLEG): container finished" podID="75d60c1c-e3ab-4514-9f51-91f56de520a9" containerID="d4e0f46f9835c3e5df4bfd3d8c95f3668a05275879fd028b86d57d256bd0888d" exitCode=60 Nov 03 22:30:04.431213 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:04.431102 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" event={"ID":"75d60c1c-e3ab-4514-9f51-91f56de520a9","Type":"ContainerDied","Data":"d4e0f46f9835c3e5df4bfd3d8c95f3668a05275879fd028b86d57d256bd0888d"} Nov 03 22:30:04.431447 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:04.431433 5084 scope.go:122] "RemoveContainer" containerID="d4e0f46f9835c3e5df4bfd3d8c95f3668a05275879fd028b86d57d256bd0888d" Nov 03 22:30:04.431732 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:04.431713 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k_odh-ai-gateway-infra(75d60c1c-e3ab-4514-9f51-91f56de520a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" podUID="75d60c1c-e3ab-4514-9f51-91f56de520a9" Nov 03 22:30:04.464163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:04.464130 5084 scope.go:122] "RemoveContainer" containerID="1143b19ceeb6b62d1cedd27f187767040cb0b8784abd0bf82aa2fa4362a72e4c" Nov 03 22:30:04.682779 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:04.682737 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-86603f" Nov 03 22:30:04.712667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:04.712607 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b262f86-3e06-4e2e-ac5f-95872e882179-kube-api-access-cvcj4\" (UniqueName: \"kubernetes.io/projected/5b262f86-3e06-4e2e-ac5f-95872e882179-kube-api-access-cvcj4\") pod \"5b262f86-3e06-4e2e-ac5f-95872e882179\" (UID: \"5b262f86-3e06-4e2e-ac5f-95872e882179\") " Nov 03 22:30:04.714793 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:04.714763 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b262f86-3e06-4e2e-ac5f-95872e882179-kube-api-access-cvcj4" pod "5b262f86-3e06-4e2e-ac5f-95872e882179" (UID: "5b262f86-3e06-4e2e-ac5f-95872e882179"). InnerVolumeSpecName "kube-api-access-cvcj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:04.814530 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:04.814493 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvcj4\" (UniqueName: \"kubernetes.io/projected/5b262f86-3e06-4e2e-ac5f-95872e882179-kube-api-access-cvcj4\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:04.944764 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:04.944658 5084 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-86603f\" is forbidden: User \"system:node:ip-10-0-141-84.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-84.us-west-2.compute.internal' and this object" podUID="5b262f86-3e06-4e2e-ac5f-95872e882179" pod="opendatahub/test-curl-7091-86603f" Nov 03 22:30:05.441690 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:05.441647 5084 generic.go:356] "Generic (PLEG): container finished" podID="5b262f86-3e06-4e2e-ac5f-95872e882179" containerID="458c279ec72d2d781e304f2b13b19ddb2a1c2598ad3449f19bca1199c6889a42" exitCode=137 Nov 03 22:30:05.441897 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:05.441761 5084 scope.go:122] "RemoveContainer" containerID="458c279ec72d2d781e304f2b13b19ddb2a1c2598ad3449f19bca1199c6889a42" Nov 03 22:30:05.441897 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:05.441894 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7091-86603f" Nov 03 22:30:05.530123 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:05.526680 5084 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-86603f\" is forbidden: User \"system:node:ip-10-0-141-84.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-84.us-west-2.compute.internal' and this object" podUID="5b262f86-3e06-4e2e-ac5f-95872e882179" pod="opendatahub/test-curl-7091-86603f" Nov 03 22:30:05.569088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:05.548210 5084 scope.go:122] "RemoveContainer" containerID="8f3d3904b8cf268aa9ada1f73413c8a9f478242b0e7e89573cc1dea213f07b4b" Nov 03 22:30:05.569088 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:05.548871 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st_odh-ai-gateway-infra(d7bf5b25-5684-439a-980c-bcb503e073b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" podUID="d7bf5b25-5684-439a-980c-bcb503e073b4" Nov 03 22:30:05.573740 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:05.573709 5084 scope.go:122] "RemoveContainer" containerID="d4e0f46f9835c3e5df4bfd3d8c95f3668a05275879fd028b86d57d256bd0888d" Nov 03 22:30:05.582803 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:05.574341 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k_odh-ai-gateway-infra(75d60c1c-e3ab-4514-9f51-91f56de520a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" podUID="75d60c1c-e3ab-4514-9f51-91f56de520a9" Nov 03 22:30:05.597821 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:05.597764 5084 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-86603f\" is forbidden: User \"system:node:ip-10-0-141-84.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-84.us-west-2.compute.internal' and this object" podUID="5b262f86-3e06-4e2e-ac5f-95872e882179" pod="opendatahub/test-curl-7091-86603f" Nov 03 22:30:05.616231 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:05.616199 5084 scope.go:122] "RemoveContainer" containerID="458c279ec72d2d781e304f2b13b19ddb2a1c2598ad3449f19bca1199c6889a42" Nov 03 22:30:05.617017 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:05.616994 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"458c279ec72d2d781e304f2b13b19ddb2a1c2598ad3449f19bca1199c6889a42\": container with ID starting with 458c279ec72d2d781e304f2b13b19ddb2a1c2598ad3449f19bca1199c6889a42 not found: ID does not exist" containerID="458c279ec72d2d781e304f2b13b19ddb2a1c2598ad3449f19bca1199c6889a42" Nov 03 22:30:05.617193 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:05.617170 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"458c279ec72d2d781e304f2b13b19ddb2a1c2598ad3449f19bca1199c6889a42"} err="failed to get container status \"458c279ec72d2d781e304f2b13b19ddb2a1c2598ad3449f19bca1199c6889a42\": rpc error: code = NotFound desc = could not find container \"458c279ec72d2d781e304f2b13b19ddb2a1c2598ad3449f19bca1199c6889a42\": container with ID starting with 458c279ec72d2d781e304f2b13b19ddb2a1c2598ad3449f19bca1199c6889a42 not found: ID does not exist" Nov 03 22:30:05.965509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:05.962431 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r"] Nov 03 22:30:05.965509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:05.963204 5084 cpu_manager.go:414] "containerMap: removing container" podUID="5b262f86-3e06-4e2e-ac5f-95872e882179" containerName="test-curl-7091-86603f" Nov 03 22:30:05.965509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:05.963221 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b262f86-3e06-4e2e-ac5f-95872e882179" containerName="test-curl-7091-86603f" Nov 03 22:30:05.965509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:05.963340 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b262f86-3e06-4e2e-ac5f-95872e882179" containerName="test-curl-7091-86603f" Nov 03 22:30:05.968854 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:05.968820 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:05.995909 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:05.995869 5084 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-86603f\" is forbidden: User \"system:node:ip-10-0-141-84.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-84.us-west-2.compute.internal' and this object" podUID="5b262f86-3e06-4e2e-ac5f-95872e882179" pod="opendatahub/test-curl-7091-86603f" Nov 03 22:30:06.016905 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.016860 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r"] Nov 03 22:30:06.022935 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.022905 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e6e7831d-openshift-default-dockercfg-ln7fq\"" Nov 03 22:30:06.048402 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:06.048340 5084 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7091-86603f\" is forbidden: User \"system:node:ip-10-0-141-84.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-84.us-west-2.compute.internal' and this object" podUID="5b262f86-3e06-4e2e-ac5f-95872e882179" pod="opendatahub/test-curl-7091-86603f" Nov 03 22:30:06.156778 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.151080 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ppwm\" (UniqueName: \"kubernetes.io/projected/3afecb8a-983a-4938-a155-c9561b7c1144-kube-api-access-6ppwm\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.156778 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.151161 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-istio-envoy\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.156778 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.151248 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-workload-certs\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.156778 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.151354 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-workload-socket\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.156778 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.151389 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afecb8a-983a-4938-a155-c9561b7c1144-istiod-ca-cert\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.156778 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.151438 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afecb8a-983a-4938-a155-c9561b7c1144-istio-podinfo\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.156778 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.151547 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-istio-data\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.156778 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.151573 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-credential-socket\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.156778 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.151601 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afecb8a-983a-4938-a155-c9561b7c1144-istio-token\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.261138 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.260826 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-istio-envoy\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.265094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.261500 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-workload-certs\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.265094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.261658 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-workload-socket\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.265094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.261707 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afecb8a-983a-4938-a155-c9561b7c1144-istiod-ca-cert\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.265094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.261756 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afecb8a-983a-4938-a155-c9561b7c1144-istio-podinfo\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.265094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.261910 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-istio-data\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.265094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.261955 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-credential-socket\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.265094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.261981 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afecb8a-983a-4938-a155-c9561b7c1144-istio-token\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.265094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.262047 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ppwm\" (UniqueName: \"kubernetes.io/projected/3afecb8a-983a-4938-a155-c9561b7c1144-kube-api-access-6ppwm\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.265094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.262970 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-workload-certs\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.265094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.263291 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-workload-socket\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.266177 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.266148 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-istio-data\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.286116 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.267396 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-credential-socket\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.297731 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.293133 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afecb8a-983a-4938-a155-c9561b7c1144-istio-podinfo\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.297731 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.294448 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afecb8a-983a-4938-a155-c9561b7c1144-istiod-ca-cert\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.317093 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.315430 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-istio-envoy\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.356171 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.352743 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afecb8a-983a-4938-a155-c9561b7c1144-istio-token\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.356171 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.353501 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ppwm\" (UniqueName: \"kubernetes.io/projected/3afecb8a-983a-4938-a155-c9561b7c1144-kube-api-access-6ppwm\") pod \"e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.595410 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.595369 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:06.630550 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.630515 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b262f86-3e06-4e2e-ac5f-95872e882179" path="/var/lib/kubelet/pods/5b262f86-3e06-4e2e-ac5f-95872e882179/volumes" Nov 03 22:30:06.802450 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.802405 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc"] Nov 03 22:30:06.809228 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.809195 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.876966 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.876168 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc"] Nov 03 22:30:06.886309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.886273 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-workload-socket\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.886571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.886550 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-data\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.886697 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.886604 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-token\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.886697 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.886638 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-credential-socket\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.886810 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.886701 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-envoy\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.886810 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.886731 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-workload-certs\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.886810 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.886780 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttlhl\" (UniqueName: \"kubernetes.io/projected/15696e53-b286-40ae-b2f2-fa6234b6426b-kube-api-access-ttlhl\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.886960 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.886891 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-podinfo\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.887013 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.886980 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15696e53-b286-40ae-b2f2-fa6234b6426b-istiod-ca-cert\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.892534 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.892503 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e3937145-openshift-default-dockercfg-7t5hk\"" Nov 03 22:30:06.987683 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.987639 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-envoy\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.987683 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.987684 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-workload-certs\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.988247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.987706 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttlhl\" (UniqueName: \"kubernetes.io/projected/15696e53-b286-40ae-b2f2-fa6234b6426b-kube-api-access-ttlhl\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.988247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.987795 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-podinfo\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.988247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.987869 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15696e53-b286-40ae-b2f2-fa6234b6426b-istiod-ca-cert\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.988247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.987941 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-workload-socket\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.988247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.987987 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-data\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.988247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.988036 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-token\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.988247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.988124 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-credential-socket\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.988247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.988191 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-workload-certs\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.988650 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.988462 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-credential-socket\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.988920 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.988888 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15696e53-b286-40ae-b2f2-fa6234b6426b-istiod-ca-cert\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.989027 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.988950 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-data\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.990144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.988413 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-workload-socket\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.990694 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.990674 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-envoy\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:06.990694 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:06.990680 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-podinfo\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:07.050791 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.050753 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttlhl\" (UniqueName: \"kubernetes.io/projected/15696e53-b286-40ae-b2f2-fa6234b6426b-kube-api-access-ttlhl\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:07.050954 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.050759 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-token\") pod \"e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:07.107077 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.107029 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r"] Nov 03 22:30:07.107618 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:30:07.107591 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afecb8a_983a_4938_a155_c9561b7c1144.slice/crio-0fee6a945376c72f7394e05d51b7a6aa6c3ce3a60b845fcebef3ee5368c766f6 WatchSource:0}: Error finding container 0fee6a945376c72f7394e05d51b7a6aa6c3ce3a60b845fcebef3ee5368c766f6: Status 404 returned error can't find the container with id 0fee6a945376c72f7394e05d51b7a6aa6c3ce3a60b845fcebef3ee5368c766f6 Nov 03 22:30:07.110334 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.110295 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:30:07.110439 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.110370 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:30:07.110439 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.110398 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:30:07.129770 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.129736 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:07.412156 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.411984 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk"] Nov 03 22:30:07.434607 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.434569 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.462289 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.462255 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk"] Nov 03 22:30:07.521411 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.521368 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.521649 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.521487 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4l4x\" (UniqueName: \"kubernetes.io/projected/cfc48515-bea2-4a71-8d48-c6000b2a46d9-kube-api-access-q4l4x\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.521649 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.521535 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.521649 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.521593 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.521649 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.521650 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.521859 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.521670 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.521859 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.521694 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.521859 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.521710 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.521859 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.521754 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.602703 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.602656 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" event={"ID":"3afecb8a-983a-4938-a155-c9561b7c1144","Type":"ContainerStarted","Data":"f444bbf9c1bcce1f1e97471f788b3d795fdbe633222732dc0675131462755ca2"} Nov 03 22:30:07.602992 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.602736 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" event={"ID":"3afecb8a-983a-4938-a155-c9561b7c1144","Type":"ContainerStarted","Data":"0fee6a945376c72f7394e05d51b7a6aa6c3ce3a60b845fcebef3ee5368c766f6"} Nov 03 22:30:07.613851 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.613806 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6"] Nov 03 22:30:07.622088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.620055 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.623554 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.623525 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.623716 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.623689 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.623783 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.623739 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.623841 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.623792 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.623841 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.623818 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.623983 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.623884 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.624038 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.623996 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.624111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.624094 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4l4x\" (UniqueName: \"kubernetes.io/projected/cfc48515-bea2-4a71-8d48-c6000b2a46d9-kube-api-access-q4l4x\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.624204 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.624168 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.624408 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.624382 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-data\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.624607 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.624585 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-credential-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.624682 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.624668 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-workload-certs\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.624925 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.624905 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-workload-socket\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.625167 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.625147 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istiod-ca-cert\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.627230 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.627203 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-envoy\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.629393 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.629368 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-podinfo\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.675334 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.674540 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc"] Nov 03 22:30:07.677885 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:30:07.676754 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15696e53_b286_40ae_b2f2_fa6234b6426b.slice/crio-16d5215754bd6c357b7b2a136592552a849fdb749b34be19cbf2e6474a781e7d WatchSource:0}: Error finding container 16d5215754bd6c357b7b2a136592552a849fdb749b34be19cbf2e6474a781e7d: Status 404 returned error can't find the container with id 16d5215754bd6c357b7b2a136592552a849fdb749b34be19cbf2e6474a781e7d Nov 03 22:30:07.687851 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.683314 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:30:07.687851 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.683424 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:30:07.687851 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.683469 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:30:07.699702 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.699594 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-token\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.710960 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.710868 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6"] Nov 03 22:30:07.710960 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.710924 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4l4x\" (UniqueName: \"kubernetes.io/projected/cfc48515-bea2-4a71-8d48-c6000b2a46d9-kube-api-access-q4l4x\") pod \"e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.732093 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.732015 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" podStartSLOduration=2.731996776 podStartE2EDuration="2.731996776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:07.727721682 +0000 UTC m=+1375.321753668" watchObservedRunningTime="2026-11-03 22:30:07.731996776 +0000 UTC m=+1375.326028750" Nov 03 22:30:07.734052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.733836 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca4f5ec-9244-447b-ba40-91409196e307-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.734052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.733870 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca4f5ec-9244-447b-ba40-91409196e307-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.734052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.733890 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt5nt\" (UniqueName: \"kubernetes.io/projected/6ca4f5ec-9244-447b-ba40-91409196e307-kube-api-access-qt5nt\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.734336 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.734058 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.734336 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.734208 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca4f5ec-9244-447b-ba40-91409196e307-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.734336 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.734262 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.734336 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.734282 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.734596 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.734437 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.734596 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.734494 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.762692 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.762654 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:07.837617 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.836410 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.837617 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.836497 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca4f5ec-9244-447b-ba40-91409196e307-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.837617 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.836524 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.837617 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.836540 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.837617 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.836596 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.837617 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.836624 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.837617 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.836694 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca4f5ec-9244-447b-ba40-91409196e307-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.837617 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.836710 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca4f5ec-9244-447b-ba40-91409196e307-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.837617 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.836729 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt5nt\" (UniqueName: \"kubernetes.io/projected/6ca4f5ec-9244-447b-ba40-91409196e307-kube-api-access-qt5nt\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.838265 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.837697 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.859166 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.838506 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca4f5ec-9244-447b-ba40-91409196e307-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.859166 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.839488 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.859166 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.840009 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.859166 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.840629 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.859166 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.855683 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca4f5ec-9244-447b-ba40-91409196e307-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:07.873086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:07.860560 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:08.047549 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.045135 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca4f5ec-9244-447b-ba40-91409196e307-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:08.049841 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.049784 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt5nt\" (UniqueName: \"kubernetes.io/projected/6ca4f5ec-9244-447b-ba40-91409196e307-kube-api-access-qt5nt\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:08.207580 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.207495 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk"] Nov 03 22:30:08.215429 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:30:08.215365 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc48515_bea2_4a71_8d48_c6000b2a46d9.slice/crio-5f6e3aa8aa4ddf736bc88ac1c72f3c1b8db45cc3b186f5a96beb05c2fc508600 WatchSource:0}: Error finding container 5f6e3aa8aa4ddf736bc88ac1c72f3c1b8db45cc3b186f5a96beb05c2fc508600: Status 404 returned error can't find the container with id 5f6e3aa8aa4ddf736bc88ac1c72f3c1b8db45cc3b186f5a96beb05c2fc508600 Nov 03 22:30:08.224102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.223463 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:30:08.224102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.223555 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:30:08.224102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.223589 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:30:08.270770 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.270738 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:08.371317 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.371274 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc"] Nov 03 22:30:08.378414 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.377480 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.451472 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.451437 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc"] Nov 03 22:30:08.472645 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.472149 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-workload-certs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.474824 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.472880 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-envoy\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.474824 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.472975 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-credential-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.474824 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.473002 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-podinfo\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.474824 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.473274 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-workload-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.474824 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.473358 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-data\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.474824 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.473384 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42sdj\" (UniqueName: \"kubernetes.io/projected/5350dd65-ca9c-4a6f-8356-9f925f84f864-kube-api-access-42sdj\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.474824 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.473418 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-token\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.474824 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.473536 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5350dd65-ca9c-4a6f-8356-9f925f84f864-istiod-ca-cert\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.574693 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.574658 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-workload-certs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.574973 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.574949 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-envoy\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.575266 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.575246 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-credential-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.575420 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.575395 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-podinfo\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.575543 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.575520 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-workload-certs\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.575614 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.575579 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-workload-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.575674 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.575663 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-data\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.575738 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.575688 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42sdj\" (UniqueName: \"kubernetes.io/projected/5350dd65-ca9c-4a6f-8356-9f925f84f864-kube-api-access-42sdj\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.575738 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.575721 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-token\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.575865 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.575844 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5350dd65-ca9c-4a6f-8356-9f925f84f864-istiod-ca-cert\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.575968 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.575928 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-credential-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.576409 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.576301 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-workload-socket\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.576726 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.576687 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5350dd65-ca9c-4a6f-8356-9f925f84f864-istiod-ca-cert\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.576899 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.576856 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-data\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.579674 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.579524 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-envoy\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.579674 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.579622 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-podinfo\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.595626 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.595525 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:08.607652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.607615 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:08.614361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.614240 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" event={"ID":"cfc48515-bea2-4a71-8d48-c6000b2a46d9","Type":"ContainerStarted","Data":"2f6c876dfb10ab128f9ce00f36b56008a8317a0fb6582897c2d2c982dbd48a8b"} Nov 03 22:30:08.615295 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.615220 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" event={"ID":"cfc48515-bea2-4a71-8d48-c6000b2a46d9","Type":"ContainerStarted","Data":"5f6e3aa8aa4ddf736bc88ac1c72f3c1b8db45cc3b186f5a96beb05c2fc508600"} Nov 03 22:30:08.618770 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.618731 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" event={"ID":"15696e53-b286-40ae-b2f2-fa6234b6426b","Type":"ContainerStarted","Data":"e8f718ea0253d6a780a8f56d489a04e9a98b07a7dd45c8306057ab353a02dfde"} Nov 03 22:30:08.618958 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.618799 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:08.618958 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.618824 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" event={"ID":"15696e53-b286-40ae-b2f2-fa6234b6426b","Type":"ContainerStarted","Data":"16d5215754bd6c357b7b2a136592552a849fdb749b34be19cbf2e6474a781e7d"} Nov 03 22:30:08.619599 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.619579 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:30:08.634374 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.634342 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-token\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.634608 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.634582 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42sdj\" (UniqueName: \"kubernetes.io/projected/5350dd65-ca9c-4a6f-8356-9f925f84f864-kube-api-access-42sdj\") pod \"e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.721325 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.721296 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:08.736195 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.736158 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6"] Nov 03 22:30:08.740571 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:30:08.740533 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca4f5ec_9244_447b_ba40_91409196e307.slice/crio-6fb1bc00dbb910478b07f6118d87cae687716636b5dbd7e8a8f45ec7458335fd WatchSource:0}: Error finding container 6fb1bc00dbb910478b07f6118d87cae687716636b5dbd7e8a8f45ec7458335fd: Status 404 returned error can't find the container with id 6fb1bc00dbb910478b07f6118d87cae687716636b5dbd7e8a8f45ec7458335fd Nov 03 22:30:08.743729 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.743679 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:30:08.743850 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.743781 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:30:08.743850 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.743822 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:30:08.763781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.763735 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:08.993683 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:08.993555 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" podStartSLOduration=2.993530738 podStartE2EDuration="2.993530738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:08.989143672 +0000 UTC m=+1376.583175641" watchObservedRunningTime="2026-11-03 22:30:08.993530738 +0000 UTC m=+1376.587562714" Nov 03 22:30:09.127094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:09.126257 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" podStartSLOduration=2.126235672 podStartE2EDuration="2.126235672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:09.116530558 +0000 UTC m=+1376.710562534" watchObservedRunningTime="2026-11-03 22:30:09.126235672 +0000 UTC m=+1376.720267650" Nov 03 22:30:09.136485 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:09.136449 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:09.160488 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:09.160453 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:09.172828 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:09.170156 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:30:09.172828 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:09.170267 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:30:09.172828 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:09.170333 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:30:09.220946 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:09.220770 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc"] Nov 03 22:30:09.631691 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:09.631649 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" event={"ID":"5350dd65-ca9c-4a6f-8356-9f925f84f864","Type":"ContainerStarted","Data":"98c682ae79b807db44a82e334201f2de3acdace88d19427e0426ef41f98f0482"} Nov 03 22:30:09.631899 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:09.631727 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" event={"ID":"5350dd65-ca9c-4a6f-8356-9f925f84f864","Type":"ContainerStarted","Data":"5845c4aab092fdd878aea6669c8eb5dc33af36a15f6061ba825e8b66ca9aced4"} Nov 03 22:30:09.634611 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:09.633907 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" event={"ID":"6ca4f5ec-9244-447b-ba40-91409196e307","Type":"ContainerStarted","Data":"976a58303b267c6211f69c2aa52cb44bfcc6c1b88751948cffedbabdcf4a64df"} Nov 03 22:30:09.634611 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:09.633956 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" event={"ID":"6ca4f5ec-9244-447b-ba40-91409196e307","Type":"ContainerStarted","Data":"6fb1bc00dbb910478b07f6118d87cae687716636b5dbd7e8a8f45ec7458335fd"} Nov 03 22:30:09.634873 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:09.634830 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:09.635756 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:09.635720 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:30:09.721664 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:09.721631 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:09.771627 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:09.771266 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:30:09.771627 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:09.771340 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 22:30:09.804541 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:09.804481 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" podStartSLOduration=1.804460685 podStartE2EDuration="1.804460685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:09.795949984 +0000 UTC m=+1377.389981960" watchObservedRunningTime="2026-11-03 22:30:09.804460685 +0000 UTC m=+1377.398492736" Nov 03 22:30:10.272533 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:10.272500 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:10.297259 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:10.297213 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 22:30:10.297555 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:10.297524 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" podUID="6ca4f5ec-9244-447b-ba40-91409196e307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 22:30:10.725503 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:10.725408 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:30:10.725767 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:10.725738 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:30:10.769634 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:10.769494 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:30:10.769634 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:10.769564 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 22:30:11.288275 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:11.281173 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 22:30:11.288275 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:11.281240 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" podUID="6ca4f5ec-9244-447b-ba40-91409196e307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 22:30:11.727158 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:11.724574 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:30:11.727158 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:11.724639 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:30:11.764094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:11.763703 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:30:11.764094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:11.763782 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 22:30:12.232097 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:12.228844 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:30:12.272563 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:12.272519 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 22:30:12.273972 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:12.273931 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" podUID="6ca4f5ec-9244-447b-ba40-91409196e307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 22:30:12.802091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:12.766241 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:30:12.932213 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:12.932161 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:30:12.933125 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:12.891771 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:30:13.082149 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:13.061169 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 22:30:13.198675 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:13.198549 5084 scope.go:122] "RemoveContainer" containerID="3891cc2c5c18701d1ab78f7997fb0466a346de77520c6f556e3dad23757aea5a" Nov 03 22:30:13.293549 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:13.292252 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 22:30:13.293549 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:13.292322 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" podUID="6ca4f5ec-9244-447b-ba40-91409196e307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 22:30:13.621720 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:13.619186 5084 scope.go:122] "RemoveContainer" containerID="c4eb8918afdc5c2a4e2c2991edd6edca0835f2916d76823630f934a54e233388" Nov 03 22:30:13.621720 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:13.619507 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:13.770319 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:13.743127 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:30:13.770604 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:13.770566 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:30:13.771116 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:13.759443 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:13.796094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:13.789321 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:30:13.796094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:13.789384 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 22:30:13.927109 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:13.905728 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 22:30:13.984348 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:13.984293 5084 scope.go:122] "RemoveContainer" containerID="b7b3b0d4429e200ccea51997af027d89a1f6c9e15d59939ef9cf3e2d72da5bc2" Nov 03 22:30:14.071278 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:14.071229 5084 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": context deadline exceeded" Nov 03 22:30:14.444512 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:14.443847 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 22:30:14.444512 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:14.443939 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" podUID="6ca4f5ec-9244-447b-ba40-91409196e307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 22:30:14.525096 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:14.519305 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:14.627407 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:14.626187 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:14.628999 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:14.627742 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:30:14.946104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:14.942372 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:15.045955 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:15.045916 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:30:15.046292 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:15.046241 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:30:15.052186 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:15.052135 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:30:15.052455 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:15.052429 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 22:30:15.059541 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:15.057534 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" podStartSLOduration=8.057512172 podStartE2EDuration="8.057512172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:10.170301922 +0000 UTC m=+1377.764333896" watchObservedRunningTime="2026-11-03 22:30:15.057512172 +0000 UTC m=+1382.651544149" Nov 03 22:30:15.242312 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:15.242257 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:30:15.330509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:15.330458 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 22:30:15.330999 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:15.330855 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" podUID="6ca4f5ec-9244-447b-ba40-91409196e307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 22:30:15.633117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:15.630592 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:15.728091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:15.724835 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:30:15.728091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:15.724915 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:30:15.764157 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:15.758991 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:30:15.768102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:15.764604 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:30:15.768102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:15.764666 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 22:30:15.886103 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:15.865211 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:30:16.605124 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:16.596220 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 22:30:16.605124 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:16.596327 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" podUID="6ca4f5ec-9244-447b-ba40-91409196e307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 22:30:16.733182 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:16.733139 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:30:16.765224 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:16.765172 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:30:16.765902 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:16.735505 5084 scope.go:122] "RemoveContainer" containerID="8f3d3904b8cf268aa9ada1f73413c8a9f478242b0e7e89573cc1dea213f07b4b" Nov 03 22:30:16.801963 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:16.801864 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:30:16.802298 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:16.802253 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 22:30:16.954795 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:16.954685 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" podUID="3b3927af-60a2-4432-99d6-b999a96f315f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 22:30:17.323258 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:17.323215 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 22:30:17.349832 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:17.349767 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" podUID="6ca4f5ec-9244-447b-ba40-91409196e307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 22:30:17.682089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:17.667176 5084 scope.go:122] "RemoveContainer" containerID="346e62c91ab2f8b54faba4ef2f1ab736cd95d9964b4a0bdd26c1b28982522cc5" Nov 03 22:30:17.682089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:17.668470 5084 scope.go:122] "RemoveContainer" containerID="43c73bf81497204d1ba90fe90f87d4b026e01968505fa302ebd47e37d48da557" Nov 03 22:30:17.752137 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:17.752093 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:30:17.752391 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:17.752359 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:30:17.863739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:17.791721 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:30:17.863739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:17.831435 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 22:30:17.863739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:17.839124 5084 scope.go:122] "RemoveContainer" containerID="d4e0f46f9835c3e5df4bfd3d8c95f3668a05275879fd028b86d57d256bd0888d" Nov 03 22:30:18.017107 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:18.017049 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" podUID="3b3927af-60a2-4432-99d6-b999a96f315f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 22:30:18.041457 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:18.041421 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:30:18.330115 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:18.322914 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 22:30:18.330115 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:18.322972 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" podUID="6ca4f5ec-9244-447b-ba40-91409196e307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 22:30:18.438130 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:18.438012 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:18.725087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:18.725008 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:30:18.742530 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:18.739769 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:18.745140 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:18.742638 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:30:18.784105 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:18.779833 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:30:18.784105 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:18.779898 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 22:30:18.864473 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:18.864423 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:19.046089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:19.042772 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" podUID="3b3927af-60a2-4432-99d6-b999a96f315f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:19.153131 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:19.128184 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:19.153131 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:19.128263 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" podUID="aa2863b4-3169-423e-a0ee-65fd49117624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:19.337090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:19.334284 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 22:30:19.337090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:19.334351 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" podUID="6ca4f5ec-9244-447b-ba40-91409196e307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 22:30:19.453721 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:19.453659 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 22:30:19.462089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:19.454455 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:30:19.740285 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:19.740169 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:19.741317 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:19.741290 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:30:19.780107 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:19.780042 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 22:30:19.807548 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:19.780400 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:30:19.807820 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:19.807783 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:30:19.950057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:19.939644 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 22:30:19.950057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:19.939728 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 22:30:20.083006 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:20.081443 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" podUID="3b3927af-60a2-4432-99d6-b999a96f315f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.312963 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:20.251704 5084 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.339080 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:20.338867 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 22:30:20.339080 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:20.338935 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" podUID="6ca4f5ec-9244-447b-ba40-91409196e307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 22:30:20.505152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:20.501851 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.862092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:20.858632 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.862092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:20.860838 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:20.932148 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:20.900783 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:30:20.932148 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:20.900848 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:30:20.990107 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:20.988032 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.430639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:21.430530 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:30:21.434256 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:21.434224 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:30:21.434579 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:21.434562 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:30:21.528269 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:21.528216 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.863224 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:21.863173 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:21.876575 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:21.869547 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:22.012861 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:22.010179 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:22.100152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:22.080659 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:30:22.326795 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:22.265141 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:30:22.326795 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:22.264999 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:22.326795 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:22.265280 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:22.372232 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:22.352306 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" podUID="3b3927af-60a2-4432-99d6-b999a96f315f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 22:30:22.372232 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:22.371688 5084 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Nov 03 22:30:22.384092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:22.382185 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:22.384092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:22.382277 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" podUID="6ca4f5ec-9244-447b-ba40-91409196e307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:22.442389 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:22.442356 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:30:22.876860 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:22.859972 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:23.257147 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:23.182390 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 22:30:23.257147 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:23.182452 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:23.257147 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:23.182532 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:30:23.257147 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:23.245910 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:30:23.272120 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:23.272053 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qb78n" podUID="bbdfd9d6-132a-423d-b96a-a0119bc8b122" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:23.320089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:23.290206 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:23.320089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:23.290298 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:23.394318 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:23.394260 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" podUID="3b3927af-60a2-4432-99d6-b999a96f315f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 22:30:23.435405 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:23.435339 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:23.435790 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:23.435750 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" podUID="6ca4f5ec-9244-447b-ba40-91409196e307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:24.101090 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:24.091870 5084 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.186s" Nov 03 22:30:24.101090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:24.091950 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:24.101090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:24.092051 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:30:24.101090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:24.092121 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:30:24.101090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:24.092139 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:24.101090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:24.092166 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:24.105196 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:24.103445 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 22:30:24.105196 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:24.103505 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 22:30:24.191263 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:24.191135 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:30:24.197846 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:24.197812 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:30:24.232095 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:24.229288 5084 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70260099_baab_410c_9f3f_9ab097b74a86.slice/crio-conmon-f5c586a1fe1b1e037748aa1d2e1dec2e7e7ae7f8f85ba510aa1af80dac89eabe.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70260099_baab_410c_9f3f_9ab097b74a86.slice/crio-f5c586a1fe1b1e037748aa1d2e1dec2e7e7ae7f8f85ba510aa1af80dac89eabe.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:30:24.319454 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:24.314670 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:24.335103 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:24.333679 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:30:24.420903 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:24.420809 5084 generic.go:356] "Generic (PLEG): container finished" podID="d7bf5b25-5684-439a-980c-bcb503e073b4" containerID="5020fede362a844f7656925ed3c701908a17beebc5bcaaceb89524ec2bf5a8e0" exitCode=60 Nov 03 22:30:24.422111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:24.422081 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" event={"ID":"d7bf5b25-5684-439a-980c-bcb503e073b4","Type":"ContainerDied","Data":"5020fede362a844f7656925ed3c701908a17beebc5bcaaceb89524ec2bf5a8e0"} Nov 03 22:30:24.422344 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:24.422326 5084 scope.go:122] "RemoveContainer" containerID="8f3d3904b8cf268aa9ada1f73413c8a9f478242b0e7e89573cc1dea213f07b4b" Nov 03 22:30:24.423323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:24.423300 5084 scope.go:122] "RemoveContainer" containerID="5020fede362a844f7656925ed3c701908a17beebc5bcaaceb89524ec2bf5a8e0" Nov 03 22:30:24.427650 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:24.427614 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st_odh-ai-gateway-infra(d7bf5b25-5684-439a-980c-bcb503e073b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" podUID="d7bf5b25-5684-439a-980c-bcb503e073b4" Nov 03 22:30:25.063487 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.063447 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:25.086233 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.086181 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:25.086433 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.086259 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:25.443273 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.443174 5084 generic.go:356] "Generic (PLEG): container finished" podID="3f65be02-e914-4d38-917e-5221c7ab028b" containerID="283bd2cc48efdc2443bd677826c2d59e8e1e4197e4f31eeaaea3e54fe751d24b" exitCode=60 Nov 03 22:30:25.443273 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.443250 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd" event={"ID":"3f65be02-e914-4d38-917e-5221c7ab028b","Type":"ContainerDied","Data":"283bd2cc48efdc2443bd677826c2d59e8e1e4197e4f31eeaaea3e54fe751d24b"} Nov 03 22:30:25.443780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.443294 5084 scope.go:122] "RemoveContainer" containerID="346e62c91ab2f8b54faba4ef2f1ab736cd95d9964b4a0bdd26c1b28982522cc5" Nov 03 22:30:25.443780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.443716 5084 scope.go:122] "RemoveContainer" containerID="283bd2cc48efdc2443bd677826c2d59e8e1e4197e4f31eeaaea3e54fe751d24b" Nov 03 22:30:25.444024 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:25.444005 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895750-gqfsd_odh-ai-gateway-infra(3f65be02-e914-4d38-917e-5221c7ab028b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd" podUID="3f65be02-e914-4d38-917e-5221c7ab028b" Nov 03 22:30:25.446584 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.446561 5084 generic.go:356] "Generic (PLEG): container finished" podID="70260099-baab-410c-9f3f-9ab097b74a86" containerID="f5c586a1fe1b1e037748aa1d2e1dec2e7e7ae7f8f85ba510aa1af80dac89eabe" exitCode=60 Nov 03 22:30:25.446686 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.446630 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g" event={"ID":"70260099-baab-410c-9f3f-9ab097b74a86","Type":"ContainerDied","Data":"f5c586a1fe1b1e037748aa1d2e1dec2e7e7ae7f8f85ba510aa1af80dac89eabe"} Nov 03 22:30:25.447024 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.447007 5084 scope.go:122] "RemoveContainer" containerID="f5c586a1fe1b1e037748aa1d2e1dec2e7e7ae7f8f85ba510aa1af80dac89eabe" Nov 03 22:30:25.447412 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:25.447391 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g_odh-ai-gateway-infra(70260099-baab-410c-9f3f-9ab097b74a86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g" podUID="70260099-baab-410c-9f3f-9ab097b74a86" Nov 03 22:30:25.448969 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.448949 5084 generic.go:356] "Generic (PLEG): container finished" podID="75d60c1c-e3ab-4514-9f51-91f56de520a9" containerID="2e34377b1a95ddbfe2f49eb43b8cd3986f9454876427fc405bd41384d6b30389" exitCode=60 Nov 03 22:30:25.449146 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.449132 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" event={"ID":"75d60c1c-e3ab-4514-9f51-91f56de520a9","Type":"ContainerDied","Data":"2e34377b1a95ddbfe2f49eb43b8cd3986f9454876427fc405bd41384d6b30389"} Nov 03 22:30:25.449481 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.449465 5084 scope.go:122] "RemoveContainer" containerID="2e34377b1a95ddbfe2f49eb43b8cd3986f9454876427fc405bd41384d6b30389" Nov 03 22:30:25.449790 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:25.449764 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k_odh-ai-gateway-infra(75d60c1c-e3ab-4514-9f51-91f56de520a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" podUID="75d60c1c-e3ab-4514-9f51-91f56de520a9" Nov 03 22:30:25.453928 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.453893 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:30:25.454043 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.453986 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:30:25.460384 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.460329 5084 scope.go:122] "RemoveContainer" containerID="43c73bf81497204d1ba90fe90f87d4b026e01968505fa302ebd47e37d48da557" Nov 03 22:30:25.502091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.501954 5084 scope.go:122] "RemoveContainer" containerID="d4e0f46f9835c3e5df4bfd3d8c95f3668a05275879fd028b86d57d256bd0888d" Nov 03 22:30:25.546872 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.546809 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:25.549298 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:25.549259 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:30:26.029923 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.029588 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st"] Nov 03 22:30:26.338179 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.338144 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" Nov 03 22:30:26.432905 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.432857 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7bf5b25-5684-439a-980c-bcb503e073b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bf5b25-5684-439a-980c-bcb503e073b4-maas-api-service-ca\") pod \"d7bf5b25-5684-439a-980c-bcb503e073b4\" (UID: \"d7bf5b25-5684-439a-980c-bcb503e073b4\") " Nov 03 22:30:26.433128 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.433028 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7bf5b25-5684-439a-980c-bcb503e073b4-kube-api-access-v6r7t\" (UniqueName: \"kubernetes.io/projected/d7bf5b25-5684-439a-980c-bcb503e073b4-kube-api-access-v6r7t\") pod \"d7bf5b25-5684-439a-980c-bcb503e073b4\" (UID: \"d7bf5b25-5684-439a-980c-bcb503e073b4\") " Nov 03 22:30:26.433338 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.433311 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7bf5b25-5684-439a-980c-bcb503e073b4-maas-api-service-ca" pod "d7bf5b25-5684-439a-980c-bcb503e073b4" (UID: "d7bf5b25-5684-439a-980c-bcb503e073b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:26.433667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.433647 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bf5b25-5684-439a-980c-bcb503e073b4-maas-api-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:26.435281 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.435251 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7bf5b25-5684-439a-980c-bcb503e073b4-kube-api-access-v6r7t" pod "d7bf5b25-5684-439a-980c-bcb503e073b4" (UID: "d7bf5b25-5684-439a-980c-bcb503e073b4"). InnerVolumeSpecName "kube-api-access-v6r7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:26.472422 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.472341 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" event={"ID":"d7bf5b25-5684-439a-980c-bcb503e073b4","Type":"ContainerDied","Data":"579393fdbb382736a8209ff7d704c486e0c9d022478e8188a29d8800b4973532"} Nov 03 22:30:26.473118 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.473093 5084 scope.go:122] "RemoveContainer" containerID="5020fede362a844f7656925ed3c701908a17beebc5bcaaceb89524ec2bf5a8e0" Nov 03 22:30:26.473389 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.472993 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st" Nov 03 22:30:26.535778 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.535743 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6r7t\" (UniqueName: \"kubernetes.io/projected/d7bf5b25-5684-439a-980c-bcb503e073b4-kube-api-access-v6r7t\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:26.586857 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.586750 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd"] Nov 03 22:30:26.656025 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.655987 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st"] Nov 03 22:30:26.662788 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.662737 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895750-742st"] Nov 03 22:30:26.824809 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.824770 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g"] Nov 03 22:30:26.877963 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.877919 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd" Nov 03 22:30:26.985094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.980397 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f65be02-e914-4d38-917e-5221c7ab028b-kube-api-access-wrclw\" (UniqueName: \"kubernetes.io/projected/3f65be02-e914-4d38-917e-5221c7ab028b-kube-api-access-wrclw\") pod \"3f65be02-e914-4d38-917e-5221c7ab028b\" (UID: \"3f65be02-e914-4d38-917e-5221c7ab028b\") " Nov 03 22:30:26.985094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.980621 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f65be02-e914-4d38-917e-5221c7ab028b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f65be02-e914-4d38-917e-5221c7ab028b-maas-api-service-ca\") pod \"3f65be02-e914-4d38-917e-5221c7ab028b\" (UID: \"3f65be02-e914-4d38-917e-5221c7ab028b\") " Nov 03 22:30:26.985094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.982678 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f65be02-e914-4d38-917e-5221c7ab028b-maas-api-service-ca" pod "3f65be02-e914-4d38-917e-5221c7ab028b" (UID: "3f65be02-e914-4d38-917e-5221c7ab028b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:26.991198 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:26.988884 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f65be02-e914-4d38-917e-5221c7ab028b-kube-api-access-wrclw" pod "3f65be02-e914-4d38-917e-5221c7ab028b" (UID: "3f65be02-e914-4d38-917e-5221c7ab028b"). InnerVolumeSpecName "kube-api-access-wrclw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:27.138417 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.135543 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f65be02-e914-4d38-917e-5221c7ab028b-maas-api-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:27.138417 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.135567 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrclw\" (UniqueName: \"kubernetes.io/projected/3f65be02-e914-4d38-917e-5221c7ab028b-kube-api-access-wrclw\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:27.146668 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.146628 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k"] Nov 03 22:30:27.156903 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.156825 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g" Nov 03 22:30:27.236718 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.236676 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70260099-baab-410c-9f3f-9ab097b74a86-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70260099-baab-410c-9f3f-9ab097b74a86-maas-api-service-ca\") pod \"70260099-baab-410c-9f3f-9ab097b74a86\" (UID: \"70260099-baab-410c-9f3f-9ab097b74a86\") " Nov 03 22:30:27.236891 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.236786 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70260099-baab-410c-9f3f-9ab097b74a86-kube-api-access-66bp4\" (UniqueName: \"kubernetes.io/projected/70260099-baab-410c-9f3f-9ab097b74a86-kube-api-access-66bp4\") pod \"70260099-baab-410c-9f3f-9ab097b74a86\" (UID: \"70260099-baab-410c-9f3f-9ab097b74a86\") " Nov 03 22:30:27.237165 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.237136 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70260099-baab-410c-9f3f-9ab097b74a86-maas-api-service-ca" pod "70260099-baab-410c-9f3f-9ab097b74a86" (UID: "70260099-baab-410c-9f3f-9ab097b74a86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:27.237473 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.237454 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70260099-baab-410c-9f3f-9ab097b74a86-maas-api-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:27.239512 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.239484 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70260099-baab-410c-9f3f-9ab097b74a86-kube-api-access-66bp4" pod "70260099-baab-410c-9f3f-9ab097b74a86" (UID: "70260099-baab-410c-9f3f-9ab097b74a86"). InnerVolumeSpecName "kube-api-access-66bp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:27.338628 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.338588 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66bp4\" (UniqueName: \"kubernetes.io/projected/70260099-baab-410c-9f3f-9ab097b74a86-kube-api-access-66bp4\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:27.364637 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.364600 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" Nov 03 22:30:27.552983 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.551992 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d60c1c-e3ab-4514-9f51-91f56de520a9-kube-api-access-5qzc8\" (UniqueName: \"kubernetes.io/projected/75d60c1c-e3ab-4514-9f51-91f56de520a9-kube-api-access-5qzc8\") pod \"75d60c1c-e3ab-4514-9f51-91f56de520a9\" (UID: \"75d60c1c-e3ab-4514-9f51-91f56de520a9\") " Nov 03 22:30:27.552983 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.552124 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75d60c1c-e3ab-4514-9f51-91f56de520a9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d60c1c-e3ab-4514-9f51-91f56de520a9-maas-api-service-ca\") pod \"75d60c1c-e3ab-4514-9f51-91f56de520a9\" (UID: \"75d60c1c-e3ab-4514-9f51-91f56de520a9\") " Nov 03 22:30:27.563386 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.553564 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d60c1c-e3ab-4514-9f51-91f56de520a9-maas-api-service-ca" pod "75d60c1c-e3ab-4514-9f51-91f56de520a9" (UID: "75d60c1c-e3ab-4514-9f51-91f56de520a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:27.573443 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.570672 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd" event={"ID":"3f65be02-e914-4d38-917e-5221c7ab028b","Type":"ContainerDied","Data":"869ece2b24e7ea1231e99443f6561b0965895650b565df650632d07fc969c4a8"} Nov 03 22:30:27.573443 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.570750 5084 scope.go:122] "RemoveContainer" containerID="283bd2cc48efdc2443bd677826c2d59e8e1e4197e4f31eeaaea3e54fe751d24b" Nov 03 22:30:27.573443 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.570860 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd" Nov 03 22:30:27.605478 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.601772 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g" event={"ID":"70260099-baab-410c-9f3f-9ab097b74a86","Type":"ContainerDied","Data":"d3613faed3e3eaeb49aa0e535835565eaedbe7d4868028545d371d30a9fff6d3"} Nov 03 22:30:27.605478 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.601909 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g" Nov 03 22:30:27.719786 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.719693 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d60c1c-e3ab-4514-9f51-91f56de520a9-kube-api-access-5qzc8" pod "75d60c1c-e3ab-4514-9f51-91f56de520a9" (UID: "75d60c1c-e3ab-4514-9f51-91f56de520a9"). InnerVolumeSpecName "kube-api-access-5qzc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:27.746086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.740822 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qzc8\" (UniqueName: \"kubernetes.io/projected/75d60c1c-e3ab-4514-9f51-91f56de520a9-kube-api-access-5qzc8\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:27.746086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.740862 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d60c1c-e3ab-4514-9f51-91f56de520a9-maas-api-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:30:27.831152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.821990 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" event={"ID":"75d60c1c-e3ab-4514-9f51-91f56de520a9","Type":"ContainerDied","Data":"9683766be29b362b2cada01748c6a3ec0f5a55da355b727dd444bbb48d60d502"} Nov 03 22:30:27.918334 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.913018 5084 scope.go:122] "RemoveContainer" containerID="f5c586a1fe1b1e037748aa1d2e1dec2e7e7ae7f8f85ba510aa1af80dac89eabe" Nov 03 22:30:27.946087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:27.942772 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k" Nov 03 22:30:28.124142 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.123613 5084 scope.go:122] "RemoveContainer" containerID="2e34377b1a95ddbfe2f49eb43b8cd3986f9454876427fc405bd41384d6b30389" Nov 03 22:30:28.297371 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.297336 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff"] Nov 03 22:30:28.299438 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.299400 5084 cpu_manager.go:414] "containerMap: removing container" podUID="3f65be02-e914-4d38-917e-5221c7ab028b" containerName="cleanup" Nov 03 22:30:28.299630 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.299616 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f65be02-e914-4d38-917e-5221c7ab028b" containerName="cleanup" Nov 03 22:30:28.299723 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.299713 5084 cpu_manager.go:414] "containerMap: removing container" podUID="d7bf5b25-5684-439a-980c-bcb503e073b4" containerName="cleanup" Nov 03 22:30:28.299817 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.299806 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7bf5b25-5684-439a-980c-bcb503e073b4" containerName="cleanup" Nov 03 22:30:28.299931 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.299921 5084 cpu_manager.go:414] "containerMap: removing container" podUID="70260099-baab-410c-9f3f-9ab097b74a86" containerName="cleanup" Nov 03 22:30:28.299998 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.299988 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70260099-baab-410c-9f3f-9ab097b74a86" containerName="cleanup" Nov 03 22:30:28.300108 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.300097 5084 cpu_manager.go:414] "containerMap: removing container" podUID="75d60c1c-e3ab-4514-9f51-91f56de520a9" containerName="cleanup" Nov 03 22:30:28.300188 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.300178 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d60c1c-e3ab-4514-9f51-91f56de520a9" containerName="cleanup" Nov 03 22:30:28.300278 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.300269 5084 cpu_manager.go:414] "containerMap: removing container" podUID="75d60c1c-e3ab-4514-9f51-91f56de520a9" containerName="cleanup" Nov 03 22:30:28.300362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.300352 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d60c1c-e3ab-4514-9f51-91f56de520a9" containerName="cleanup" Nov 03 22:30:28.300455 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.300445 5084 cpu_manager.go:414] "containerMap: removing container" podUID="d7bf5b25-5684-439a-980c-bcb503e073b4" containerName="cleanup" Nov 03 22:30:28.300521 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.300512 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7bf5b25-5684-439a-980c-bcb503e073b4" containerName="cleanup" Nov 03 22:30:28.300596 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.300587 5084 cpu_manager.go:414] "containerMap: removing container" podUID="3f65be02-e914-4d38-917e-5221c7ab028b" containerName="cleanup" Nov 03 22:30:28.300662 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.300652 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f65be02-e914-4d38-917e-5221c7ab028b" containerName="cleanup" Nov 03 22:30:28.300784 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.300775 5084 cpu_manager.go:414] "containerMap: removing container" podUID="70260099-baab-410c-9f3f-9ab097b74a86" containerName="cleanup" Nov 03 22:30:28.300859 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.300850 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70260099-baab-410c-9f3f-9ab097b74a86" containerName="cleanup" Nov 03 22:30:28.300987 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.300978 5084 cpu_manager.go:414] "containerMap: removing container" podUID="d7bf5b25-5684-439a-980c-bcb503e073b4" containerName="cleanup" Nov 03 22:30:28.301053 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.301043 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7bf5b25-5684-439a-980c-bcb503e073b4" containerName="cleanup" Nov 03 22:30:28.301656 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.301630 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7bf5b25-5684-439a-980c-bcb503e073b4" containerName="cleanup" Nov 03 22:30:28.301833 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.301818 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f65be02-e914-4d38-917e-5221c7ab028b" containerName="cleanup" Nov 03 22:30:28.301929 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.301919 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="70260099-baab-410c-9f3f-9ab097b74a86" containerName="cleanup" Nov 03 22:30:28.311368 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.311332 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d60c1c-e3ab-4514-9f51-91f56de520a9" containerName="cleanup" Nov 03 22:30:28.311635 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.311621 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7bf5b25-5684-439a-980c-bcb503e073b4" containerName="cleanup" Nov 03 22:30:28.311725 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.311716 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f65be02-e914-4d38-917e-5221c7ab028b" containerName="cleanup" Nov 03 22:30:28.311801 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.311792 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7bf5b25-5684-439a-980c-bcb503e073b4" containerName="cleanup" Nov 03 22:30:28.311887 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.311876 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="70260099-baab-410c-9f3f-9ab097b74a86" containerName="cleanup" Nov 03 22:30:28.311966 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.311957 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d60c1c-e3ab-4514-9f51-91f56de520a9" containerName="cleanup" Nov 03 22:30:28.312035 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.312027 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d60c1c-e3ab-4514-9f51-91f56de520a9" containerName="cleanup" Nov 03 22:30:28.336597 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.335941 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff"] Nov 03 22:30:28.336597 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.336011 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj"] Nov 03 22:30:28.337354 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.337168 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff" Nov 03 22:30:28.337577 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.337551 5084 cpu_manager.go:414] "containerMap: removing container" podUID="3f65be02-e914-4d38-917e-5221c7ab028b" containerName="cleanup" Nov 03 22:30:28.337656 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.337580 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f65be02-e914-4d38-917e-5221c7ab028b" containerName="cleanup" Nov 03 22:30:28.337656 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.337642 5084 cpu_manager.go:414] "containerMap: removing container" podUID="75d60c1c-e3ab-4514-9f51-91f56de520a9" containerName="cleanup" Nov 03 22:30:28.337656 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.337652 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d60c1c-e3ab-4514-9f51-91f56de520a9" containerName="cleanup" Nov 03 22:30:28.337827 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.337709 5084 cpu_manager.go:414] "containerMap: removing container" podUID="70260099-baab-410c-9f3f-9ab097b74a86" containerName="cleanup" Nov 03 22:30:28.337827 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.337716 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70260099-baab-410c-9f3f-9ab097b74a86" containerName="cleanup" Nov 03 22:30:28.341338 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.341305 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f65be02-e914-4d38-917e-5221c7ab028b" containerName="cleanup" Nov 03 22:30:28.341595 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.341576 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="70260099-baab-410c-9f3f-9ab097b74a86" containerName="cleanup" Nov 03 22:30:28.349922 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.349890 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj"] Nov 03 22:30:28.350287 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.350268 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj" Nov 03 22:30:28.383173 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.376726 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37bdbda-c89d-43bf-84a7-6940116bc31f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff\" (UID: \"a37bdbda-c89d-43bf-84a7-6940116bc31f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff" Nov 03 22:30:28.383173 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.376777 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2chz\" (UniqueName: \"kubernetes.io/projected/a37bdbda-c89d-43bf-84a7-6940116bc31f-kube-api-access-s2chz\") pod \"payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff\" (UID: \"a37bdbda-c89d-43bf-84a7-6940116bc31f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff" Nov 03 22:30:28.383435 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.383402 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd"] Nov 03 22:30:28.387306 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.386874 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-gqfsd"] Nov 03 22:30:28.397515 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.397481 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k"] Nov 03 22:30:28.399524 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.399391 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-cee4c09d-dockercfg-6hxjd\"" Nov 03 22:30:28.399524 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.399484 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-cee4c09d\"" Nov 03 22:30:28.401569 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.401541 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3378fd-29895750-vfc4k"] Nov 03 22:30:28.478173 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.478123 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37bdbda-c89d-43bf-84a7-6940116bc31f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff\" (UID: \"a37bdbda-c89d-43bf-84a7-6940116bc31f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff" Nov 03 22:30:28.478357 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.478185 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5fqn\" (UniqueName: \"kubernetes.io/projected/62161443-27f9-4dab-b2e1-b4616adf2374-kube-api-access-n5fqn\") pod \"payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj\" (UID: \"62161443-27f9-4dab-b2e1-b4616adf2374\") " pod="openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj" Nov 03 22:30:28.478357 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.478226 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62161443-27f9-4dab-b2e1-b4616adf2374-plugins-config-volume\") pod \"payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj\" (UID: \"62161443-27f9-4dab-b2e1-b4616adf2374\") " pod="openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj" Nov 03 22:30:28.478357 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.478264 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2chz\" (UniqueName: \"kubernetes.io/projected/a37bdbda-c89d-43bf-84a7-6940116bc31f-kube-api-access-s2chz\") pod \"payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff\" (UID: \"a37bdbda-c89d-43bf-84a7-6940116bc31f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff" Nov 03 22:30:28.478885 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.478861 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37bdbda-c89d-43bf-84a7-6940116bc31f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff\" (UID: \"a37bdbda-c89d-43bf-84a7-6940116bc31f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff" Nov 03 22:30:28.493057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.493017 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g"] Nov 03 22:30:28.497389 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.497352 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f81f8f-29895750-s7r4g"] Nov 03 22:30:28.537004 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.536961 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2chz\" (UniqueName: \"kubernetes.io/projected/a37bdbda-c89d-43bf-84a7-6940116bc31f-kube-api-access-s2chz\") pod \"payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff\" (UID: \"a37bdbda-c89d-43bf-84a7-6940116bc31f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff" Nov 03 22:30:28.580523 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.580469 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5fqn\" (UniqueName: \"kubernetes.io/projected/62161443-27f9-4dab-b2e1-b4616adf2374-kube-api-access-n5fqn\") pod \"payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj\" (UID: \"62161443-27f9-4dab-b2e1-b4616adf2374\") " pod="openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj" Nov 03 22:30:28.580995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.580536 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62161443-27f9-4dab-b2e1-b4616adf2374-plugins-config-volume\") pod \"payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj\" (UID: \"62161443-27f9-4dab-b2e1-b4616adf2374\") " pod="openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj" Nov 03 22:30:28.581590 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.581561 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62161443-27f9-4dab-b2e1-b4616adf2374-plugins-config-volume\") pod \"payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj\" (UID: \"62161443-27f9-4dab-b2e1-b4616adf2374\") " pod="openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj" Nov 03 22:30:28.631163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.631130 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f65be02-e914-4d38-917e-5221c7ab028b" path="/var/lib/kubelet/pods/3f65be02-e914-4d38-917e-5221c7ab028b/volumes" Nov 03 22:30:28.631650 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.631634 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70260099-baab-410c-9f3f-9ab097b74a86" path="/var/lib/kubelet/pods/70260099-baab-410c-9f3f-9ab097b74a86/volumes" Nov 03 22:30:28.632109 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.632092 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d60c1c-e3ab-4514-9f51-91f56de520a9" path="/var/lib/kubelet/pods/75d60c1c-e3ab-4514-9f51-91f56de520a9/volumes" Nov 03 22:30:28.632844 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.632829 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7bf5b25-5684-439a-980c-bcb503e073b4" path="/var/lib/kubelet/pods/d7bf5b25-5684-439a-980c-bcb503e073b4/volumes" Nov 03 22:30:28.638374 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.638301 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5fqn\" (UniqueName: \"kubernetes.io/projected/62161443-27f9-4dab-b2e1-b4616adf2374-kube-api-access-n5fqn\") pod \"payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj\" (UID: \"62161443-27f9-4dab-b2e1-b4616adf2374\") " pod="openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj" Nov 03 22:30:28.701190 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.701148 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff" Nov 03 22:30:28.704739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:28.704605 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj" Nov 03 22:30:29.098925 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:29.098882 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff"] Nov 03 22:30:29.099364 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:30:29.099332 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37bdbda_c89d_43bf_84a7_6940116bc31f.slice/crio-b477b566fb0188c587362df598693e3c0d268b9a61397d5514129321be6eea3e WatchSource:0}: Error finding container b477b566fb0188c587362df598693e3c0d268b9a61397d5514129321be6eea3e: Status 404 returned error can't find the container with id b477b566fb0188c587362df598693e3c0d268b9a61397d5514129321be6eea3e Nov 03 22:30:29.148466 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:29.148430 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj"] Nov 03 22:30:29.148716 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:30:29.148691 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62161443_27f9_4dab_b2e1_b4616adf2374.slice/crio-23c9622914a2994c14248309109ca7ae4486e6af26ebc48b94a8026a41062448 WatchSource:0}: Error finding container 23c9622914a2994c14248309109ca7ae4486e6af26ebc48b94a8026a41062448: Status 404 returned error can't find the container with id 23c9622914a2994c14248309109ca7ae4486e6af26ebc48b94a8026a41062448 Nov 03 22:30:29.851787 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:29.851745 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj" event={"ID":"62161443-27f9-4dab-b2e1-b4616adf2374","Type":"ContainerStarted","Data":"bb4dfa2197473e7c5e08a4f203428403a75fb91bf7fb89c8a73124910a1458ba"} Nov 03 22:30:29.852302 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:29.851817 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj" event={"ID":"62161443-27f9-4dab-b2e1-b4616adf2374","Type":"ContainerStarted","Data":"23c9622914a2994c14248309109ca7ae4486e6af26ebc48b94a8026a41062448"} Nov 03 22:30:29.852302 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:29.851849 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj" Nov 03 22:30:29.853594 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:29.853566 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff" event={"ID":"a37bdbda-c89d-43bf-84a7-6940116bc31f","Type":"ContainerStarted","Data":"8540340c764eb794741e842de71c3d2facc153943d06327bf234aa01babcd46b"} Nov 03 22:30:29.853704 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:29.853628 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff" Nov 03 22:30:29.853704 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:29.853653 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff" event={"ID":"a37bdbda-c89d-43bf-84a7-6940116bc31f","Type":"ContainerStarted","Data":"b477b566fb0188c587362df598693e3c0d268b9a61397d5514129321be6eea3e"} Nov 03 22:30:29.973379 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:29.973317 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj" podStartSLOduration=2.97329989 podStartE2EDuration="2.97329989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:29.969442118 +0000 UTC m=+1397.563474091" watchObservedRunningTime="2026-11-03 22:30:29.97329989 +0000 UTC m=+1397.567331863" Nov 03 22:30:30.090246 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:30.090183 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff" podStartSLOduration=3.090160314 podStartE2EDuration="3.090160314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:30.088503563 +0000 UTC m=+1397.682535538" watchObservedRunningTime="2026-11-03 22:30:30.090160314 +0000 UTC m=+1397.684192290" Nov 03 22:30:30.523089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:30.515411 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb"] Nov 03 22:30:30.545095 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:30.533448 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb" Nov 03 22:30:30.581678 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:30.581625 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb"] Nov 03 22:30:30.592485 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:30.588842 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e6e7831d\"" Nov 03 22:30:30.592485 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:30.589572 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e6e7831d-dockercfg-kndkz\"" Nov 03 22:30:31.007962 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.004528 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll74q\" (UniqueName: \"kubernetes.io/projected/8ea29e4c-d210-48c0-b495-45958a542245-kube-api-access-ll74q\") pod \"payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb\" (UID: \"8ea29e4c-d210-48c0-b495-45958a542245\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb" Nov 03 22:30:31.009083 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.008858 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea29e4c-d210-48c0-b495-45958a542245-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb\" (UID: \"8ea29e4c-d210-48c0-b495-45958a542245\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb" Nov 03 22:30:31.050705 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.050672 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh"] Nov 03 22:30:31.058656 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.058620 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh" Nov 03 22:30:31.071951 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.071916 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh"] Nov 03 22:30:31.110408 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.110366 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll74q\" (UniqueName: \"kubernetes.io/projected/8ea29e4c-d210-48c0-b495-45958a542245-kube-api-access-ll74q\") pod \"payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb\" (UID: \"8ea29e4c-d210-48c0-b495-45958a542245\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb" Nov 03 22:30:31.110608 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.110459 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea29e4c-d210-48c0-b495-45958a542245-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb\" (UID: \"8ea29e4c-d210-48c0-b495-45958a542245\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb" Nov 03 22:30:31.111266 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.111244 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea29e4c-d210-48c0-b495-45958a542245-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb\" (UID: \"8ea29e4c-d210-48c0-b495-45958a542245\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb" Nov 03 22:30:31.170931 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.170889 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll74q\" (UniqueName: \"kubernetes.io/projected/8ea29e4c-d210-48c0-b495-45958a542245-kube-api-access-ll74q\") pod \"payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb\" (UID: \"8ea29e4c-d210-48c0-b495-45958a542245\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb" Nov 03 22:30:31.209753 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.209710 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb" Nov 03 22:30:31.212577 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.212527 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c7cdcde-5c77-491a-b84e-5f285ec4d794-plugins-config-volume\") pod \"payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh\" (UID: \"8c7cdcde-5c77-491a-b84e-5f285ec4d794\") " pod="openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh" Nov 03 22:30:31.212763 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.212589 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mksq\" (UniqueName: \"kubernetes.io/projected/8c7cdcde-5c77-491a-b84e-5f285ec4d794-kube-api-access-4mksq\") pod \"payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh\" (UID: \"8c7cdcde-5c77-491a-b84e-5f285ec4d794\") " pod="openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh" Nov 03 22:30:31.314247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.314201 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c7cdcde-5c77-491a-b84e-5f285ec4d794-plugins-config-volume\") pod \"payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh\" (UID: \"8c7cdcde-5c77-491a-b84e-5f285ec4d794\") " pod="openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh" Nov 03 22:30:31.314423 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.314295 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mksq\" (UniqueName: \"kubernetes.io/projected/8c7cdcde-5c77-491a-b84e-5f285ec4d794-kube-api-access-4mksq\") pod \"payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh\" (UID: \"8c7cdcde-5c77-491a-b84e-5f285ec4d794\") " pod="openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh" Nov 03 22:30:31.314914 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.314894 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c7cdcde-5c77-491a-b84e-5f285ec4d794-plugins-config-volume\") pod \"payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh\" (UID: \"8c7cdcde-5c77-491a-b84e-5f285ec4d794\") " pod="openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh" Nov 03 22:30:31.373238 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.373200 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mksq\" (UniqueName: \"kubernetes.io/projected/8c7cdcde-5c77-491a-b84e-5f285ec4d794-kube-api-access-4mksq\") pod \"payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh\" (UID: \"8c7cdcde-5c77-491a-b84e-5f285ec4d794\") " pod="openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh" Nov 03 22:30:31.388355 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.388319 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh" Nov 03 22:30:31.602297 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:30:31.602257 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea29e4c_d210_48c0_b495_45958a542245.slice/crio-ba3ee7b362fdf13349d1844d6f78e146e039b14fa5d32bb6bd2d2917c25b7879 WatchSource:0}: Error finding container ba3ee7b362fdf13349d1844d6f78e146e039b14fa5d32bb6bd2d2917c25b7879: Status 404 returned error can't find the container with id ba3ee7b362fdf13349d1844d6f78e146e039b14fa5d32bb6bd2d2917c25b7879 Nov 03 22:30:31.603913 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.603882 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb"] Nov 03 22:30:31.775931 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:31.775897 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh"] Nov 03 22:30:31.776468 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:30:31.776437 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7cdcde_5c77_491a_b84e_5f285ec4d794.slice/crio-dfb53689e4c54e1ebd3458532a427213844a9b5d87cf78b52d41fcb217c9d412 WatchSource:0}: Error finding container dfb53689e4c54e1ebd3458532a427213844a9b5d87cf78b52d41fcb217c9d412: Status 404 returned error can't find the container with id dfb53689e4c54e1ebd3458532a427213844a9b5d87cf78b52d41fcb217c9d412 Nov 03 22:30:32.067140 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:32.067099 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb" event={"ID":"8ea29e4c-d210-48c0-b495-45958a542245","Type":"ContainerStarted","Data":"72585f2eb300fc71097746ee60ff826a2769c733c1c7a832db70c1315de9c88e"} Nov 03 22:30:32.067639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:32.067170 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb" event={"ID":"8ea29e4c-d210-48c0-b495-45958a542245","Type":"ContainerStarted","Data":"ba3ee7b362fdf13349d1844d6f78e146e039b14fa5d32bb6bd2d2917c25b7879"} Nov 03 22:30:32.067639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:32.067200 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb" Nov 03 22:30:32.068909 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:32.068881 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh" event={"ID":"8c7cdcde-5c77-491a-b84e-5f285ec4d794","Type":"ContainerStarted","Data":"16d5f1104c1afd90a30c8f9ca2aa32763eb2c2bf3bf64757142b3cdb615ce73b"} Nov 03 22:30:32.068976 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:32.068939 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh" Nov 03 22:30:32.068976 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:32.068964 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh" event={"ID":"8c7cdcde-5c77-491a-b84e-5f285ec4d794","Type":"ContainerStarted","Data":"dfb53689e4c54e1ebd3458532a427213844a9b5d87cf78b52d41fcb217c9d412"} Nov 03 22:30:32.184787 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:32.184735 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb" podStartSLOduration=2.184713389 podStartE2EDuration="2.184713389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:32.183121676 +0000 UTC m=+1399.777153649" watchObservedRunningTime="2026-11-03 22:30:32.184713389 +0000 UTC m=+1399.778745363" Nov 03 22:30:33.305040 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.298410 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh" podStartSLOduration=3.298385977 podStartE2EDuration="3.298385977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:32.31384186 +0000 UTC m=+1399.907873835" watchObservedRunningTime="2026-11-03 22:30:33.298385977 +0000 UTC m=+1400.892417955" Nov 03 22:30:33.305040 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.299012 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2"] Nov 03 22:30:33.357607 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.357574 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2"] Nov 03 22:30:33.358049 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.358027 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" Nov 03 22:30:33.539214 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.534214 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-59a07a45-dockercfg-zxtrj\"" Nov 03 22:30:33.539214 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.534368 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-59a07a45\"" Nov 03 22:30:33.561144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.540548 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5fce82-2002-48a4-ba14-e5e4bf6bfed5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2\" (UID: \"2a5fce82-2002-48a4-ba14-e5e4bf6bfed5\") " pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" Nov 03 22:30:33.561144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.540749 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlqk9\" (UniqueName: \"kubernetes.io/projected/2a5fce82-2002-48a4-ba14-e5e4bf6bfed5-kube-api-access-nlqk9\") pod \"payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2\" (UID: \"2a5fce82-2002-48a4-ba14-e5e4bf6bfed5\") " pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" Nov 03 22:30:33.596123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.587233 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj"] Nov 03 22:30:33.605742 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.604387 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" Nov 03 22:30:33.611155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.608592 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj"] Nov 03 22:30:33.641685 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.641640 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlqk9\" (UniqueName: \"kubernetes.io/projected/2a5fce82-2002-48a4-ba14-e5e4bf6bfed5-kube-api-access-nlqk9\") pod \"payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2\" (UID: \"2a5fce82-2002-48a4-ba14-e5e4bf6bfed5\") " pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" Nov 03 22:30:33.641858 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.641793 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g5pn\" (UniqueName: \"kubernetes.io/projected/63b8796a-e0c0-491c-a294-0b9eb689639c-kube-api-access-4g5pn\") pod \"payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj\" (UID: \"63b8796a-e0c0-491c-a294-0b9eb689639c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" Nov 03 22:30:33.641937 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.641922 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5fce82-2002-48a4-ba14-e5e4bf6bfed5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2\" (UID: \"2a5fce82-2002-48a4-ba14-e5e4bf6bfed5\") " pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" Nov 03 22:30:33.642046 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.641977 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b8796a-e0c0-491c-a294-0b9eb689639c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj\" (UID: \"63b8796a-e0c0-491c-a294-0b9eb689639c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" Nov 03 22:30:33.642731 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.642705 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5fce82-2002-48a4-ba14-e5e4bf6bfed5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2\" (UID: \"2a5fce82-2002-48a4-ba14-e5e4bf6bfed5\") " pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" Nov 03 22:30:33.699954 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.699921 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlqk9\" (UniqueName: \"kubernetes.io/projected/2a5fce82-2002-48a4-ba14-e5e4bf6bfed5-kube-api-access-nlqk9\") pod \"payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2\" (UID: \"2a5fce82-2002-48a4-ba14-e5e4bf6bfed5\") " pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" Nov 03 22:30:33.742959 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.742920 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g5pn\" (UniqueName: \"kubernetes.io/projected/63b8796a-e0c0-491c-a294-0b9eb689639c-kube-api-access-4g5pn\") pod \"payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj\" (UID: \"63b8796a-e0c0-491c-a294-0b9eb689639c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" Nov 03 22:30:33.743186 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.743057 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b8796a-e0c0-491c-a294-0b9eb689639c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj\" (UID: \"63b8796a-e0c0-491c-a294-0b9eb689639c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" Nov 03 22:30:33.743827 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.743799 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b8796a-e0c0-491c-a294-0b9eb689639c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj\" (UID: \"63b8796a-e0c0-491c-a294-0b9eb689639c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" Nov 03 22:30:33.802957 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.802924 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g5pn\" (UniqueName: \"kubernetes.io/projected/63b8796a-e0c0-491c-a294-0b9eb689639c-kube-api-access-4g5pn\") pod \"payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj\" (UID: \"63b8796a-e0c0-491c-a294-0b9eb689639c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" Nov 03 22:30:33.907220 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.907130 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" Nov 03 22:30:33.934797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:33.934760 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" Nov 03 22:30:34.325122 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:34.320838 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2"] Nov 03 22:30:34.344247 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:30:34.344203 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5fce82_2002_48a4_ba14_e5e4bf6bfed5.slice/crio-2db1e866c88ffd89f97c6cb1052fd83c2ffd1347fc2d75eb78d3e526bbc71829 WatchSource:0}: Error finding container 2db1e866c88ffd89f97c6cb1052fd83c2ffd1347fc2d75eb78d3e526bbc71829: Status 404 returned error can't find the container with id 2db1e866c88ffd89f97c6cb1052fd83c2ffd1347fc2d75eb78d3e526bbc71829 Nov 03 22:30:34.359618 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:34.359581 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj"] Nov 03 22:30:34.361288 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:30:34.361254 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b8796a_e0c0_491c_a294_0b9eb689639c.slice/crio-6eb2ae439539c15faf8b8b18324de5e490401805bbf3fe3d534d105bfdba2dec WatchSource:0}: Error finding container 6eb2ae439539c15faf8b8b18324de5e490401805bbf3fe3d534d105bfdba2dec: Status 404 returned error can't find the container with id 6eb2ae439539c15faf8b8b18324de5e490401805bbf3fe3d534d105bfdba2dec Nov 03 22:30:35.112574 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:35.112528 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" event={"ID":"63b8796a-e0c0-491c-a294-0b9eb689639c","Type":"ContainerStarted","Data":"87b74282896b09eafe11ccc52a8d4ab6de5236b6d753c64186c8e2cd2682e90e"} Nov 03 22:30:35.112760 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:35.112593 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" event={"ID":"63b8796a-e0c0-491c-a294-0b9eb689639c","Type":"ContainerStarted","Data":"6eb2ae439539c15faf8b8b18324de5e490401805bbf3fe3d534d105bfdba2dec"} Nov 03 22:30:35.112760 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:35.112626 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" Nov 03 22:30:35.114428 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:35.114402 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" event={"ID":"2a5fce82-2002-48a4-ba14-e5e4bf6bfed5","Type":"ContainerStarted","Data":"e61ffeac612291c8923b5bc0f444ff879f9ec99980ef7786565523779afb3427"} Nov 03 22:30:35.114572 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:35.114445 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" event={"ID":"2a5fce82-2002-48a4-ba14-e5e4bf6bfed5","Type":"ContainerStarted","Data":"2db1e866c88ffd89f97c6cb1052fd83c2ffd1347fc2d75eb78d3e526bbc71829"} Nov 03 22:30:35.114572 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:35.114481 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" Nov 03 22:30:35.229337 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:35.229269 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" podStartSLOduration=2.229247885 podStartE2EDuration="2.229247885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:35.228437447 +0000 UTC m=+1402.822469441" watchObservedRunningTime="2026-11-03 22:30:35.229247885 +0000 UTC m=+1402.823279860" Nov 03 22:30:38.447786 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:38.447687 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" podStartSLOduration=5.44766262 podStartE2EDuration="5.44766262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:35.381827883 +0000 UTC m=+1402.975859858" watchObservedRunningTime="2026-11-03 22:30:38.44766262 +0000 UTC m=+1406.041694608" Nov 03 22:30:38.472086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:38.458722 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq"] Nov 03 22:30:38.472086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:38.465801 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq" Nov 03 22:30:38.479712 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:38.479669 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec48958-96f5-4e49-a495-df51c0c5c029-plugins-config-volume\") pod \"payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq\" (UID: \"1ec48958-96f5-4e49-a495-df51c0c5c029\") " pod="openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq" Nov 03 22:30:38.480221 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:38.480195 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgw8v\" (UniqueName: \"kubernetes.io/projected/1ec48958-96f5-4e49-a495-df51c0c5c029-kube-api-access-fgw8v\") pod \"payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq\" (UID: \"1ec48958-96f5-4e49-a495-df51c0c5c029\") " pod="openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq" Nov 03 22:30:38.517215 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:38.515305 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq"] Nov 03 22:30:38.567347 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:38.563724 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-e3937145\"" Nov 03 22:30:38.567347 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:38.564503 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-e3937145-dockercfg-b5944\"" Nov 03 22:30:38.672235 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:38.672154 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgw8v\" (UniqueName: \"kubernetes.io/projected/1ec48958-96f5-4e49-a495-df51c0c5c029-kube-api-access-fgw8v\") pod \"payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq\" (UID: \"1ec48958-96f5-4e49-a495-df51c0c5c029\") " pod="openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq" Nov 03 22:30:38.672661 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:38.672633 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec48958-96f5-4e49-a495-df51c0c5c029-plugins-config-volume\") pod \"payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq\" (UID: \"1ec48958-96f5-4e49-a495-df51c0c5c029\") " pod="openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq" Nov 03 22:30:38.701849 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:38.694897 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec48958-96f5-4e49-a495-df51c0c5c029-plugins-config-volume\") pod \"payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq\" (UID: \"1ec48958-96f5-4e49-a495-df51c0c5c029\") " pod="openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq" Nov 03 22:30:38.930765 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:38.930723 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgw8v\" (UniqueName: \"kubernetes.io/projected/1ec48958-96f5-4e49-a495-df51c0c5c029-kube-api-access-fgw8v\") pod \"payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq\" (UID: \"1ec48958-96f5-4e49-a495-df51c0c5c029\") " pod="openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq" Nov 03 22:30:38.942563 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:38.942490 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8"] Nov 03 22:30:38.962201 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:38.962111 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8"] Nov 03 22:30:38.962378 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:38.962269 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8" Nov 03 22:30:39.093705 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:39.093662 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb841e6-f440-4cf4-a36b-4604c2bc92c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8\" (UID: \"2eb841e6-f440-4cf4-a36b-4604c2bc92c3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8" Nov 03 22:30:39.093902 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:39.093820 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5x29\" (UniqueName: \"kubernetes.io/projected/2eb841e6-f440-4cf4-a36b-4604c2bc92c3-kube-api-access-q5x29\") pod \"payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8\" (UID: \"2eb841e6-f440-4cf4-a36b-4604c2bc92c3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8" Nov 03 22:30:39.107275 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:39.107225 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq" Nov 03 22:30:39.197333 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:39.196682 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb841e6-f440-4cf4-a36b-4604c2bc92c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8\" (UID: \"2eb841e6-f440-4cf4-a36b-4604c2bc92c3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8" Nov 03 22:30:39.197333 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:39.196805 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5x29\" (UniqueName: \"kubernetes.io/projected/2eb841e6-f440-4cf4-a36b-4604c2bc92c3-kube-api-access-q5x29\") pod \"payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8\" (UID: \"2eb841e6-f440-4cf4-a36b-4604c2bc92c3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8" Nov 03 22:30:39.198163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:39.198119 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb841e6-f440-4cf4-a36b-4604c2bc92c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8\" (UID: \"2eb841e6-f440-4cf4-a36b-4604c2bc92c3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8" Nov 03 22:30:39.256243 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:39.256139 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5x29\" (UniqueName: \"kubernetes.io/projected/2eb841e6-f440-4cf4-a36b-4604c2bc92c3-kube-api-access-q5x29\") pod \"payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8\" (UID: \"2eb841e6-f440-4cf4-a36b-4604c2bc92c3\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8" Nov 03 22:30:39.289224 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:39.289184 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8" Nov 03 22:30:39.498625 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:39.498561 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq"] Nov 03 22:30:39.500087 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:30:39.500036 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec48958_96f5_4e49_a495_df51c0c5c029.slice/crio-e9e44ba0766ec423dd53aa27c2526a2415461449edbb62d157bee0583d3ac3aa WatchSource:0}: Error finding container e9e44ba0766ec423dd53aa27c2526a2415461449edbb62d157bee0583d3ac3aa: Status 404 returned error can't find the container with id e9e44ba0766ec423dd53aa27c2526a2415461449edbb62d157bee0583d3ac3aa Nov 03 22:30:39.680635 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:39.680599 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8"] Nov 03 22:30:39.681336 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:30:39.681308 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb841e6_f440_4cf4_a36b_4604c2bc92c3.slice/crio-13795cdb6052bf623664b1c9c4b774728b8efdee04e6988e1ea9cd2e24aeb66b WatchSource:0}: Error finding container 13795cdb6052bf623664b1c9c4b774728b8efdee04e6988e1ea9cd2e24aeb66b: Status 404 returned error can't find the container with id 13795cdb6052bf623664b1c9c4b774728b8efdee04e6988e1ea9cd2e24aeb66b Nov 03 22:30:40.217347 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:40.217307 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq" event={"ID":"1ec48958-96f5-4e49-a495-df51c0c5c029","Type":"ContainerStarted","Data":"48a23feeba821c854b473564da943e892b808285208e8783d121980f43e23b4f"} Nov 03 22:30:40.217564 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:40.217376 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq" event={"ID":"1ec48958-96f5-4e49-a495-df51c0c5c029","Type":"ContainerStarted","Data":"e9e44ba0766ec423dd53aa27c2526a2415461449edbb62d157bee0583d3ac3aa"} Nov 03 22:30:40.217564 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:40.217408 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq" Nov 03 22:30:40.219452 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:40.219420 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8" event={"ID":"2eb841e6-f440-4cf4-a36b-4604c2bc92c3","Type":"ContainerStarted","Data":"72eb3968e9ed633b63cf718c2295b4715f159289acb1383f8943a037368530a7"} Nov 03 22:30:40.219599 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:40.219470 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8" event={"ID":"2eb841e6-f440-4cf4-a36b-4604c2bc92c3","Type":"ContainerStarted","Data":"13795cdb6052bf623664b1c9c4b774728b8efdee04e6988e1ea9cd2e24aeb66b"} Nov 03 22:30:40.219599 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:40.219519 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8" Nov 03 22:30:40.339028 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:40.338964 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq" podStartSLOduration=2.338943979 podStartE2EDuration="2.338943979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:40.33451752 +0000 UTC m=+1407.928549486" watchObservedRunningTime="2026-11-03 22:30:40.338943979 +0000 UTC m=+1407.932975992" Nov 03 22:30:40.511151 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:40.510957 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8" podStartSLOduration=2.510933958 podStartE2EDuration="2.510933958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:40.486438999 +0000 UTC m=+1408.080470974" watchObservedRunningTime="2026-11-03 22:30:40.510933958 +0000 UTC m=+1408.104965934" Nov 03 22:30:41.061896 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:41.061861 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj" Nov 03 22:30:41.062556 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:41.062536 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff" Nov 03 22:30:42.153649 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:42.153607 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:30:43.094367 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:43.094336 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh" Nov 03 22:30:43.094570 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:43.094410 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb" Nov 03 22:30:46.128278 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:46.128243 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" Nov 03 22:30:46.128699 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:46.128317 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" Nov 03 22:30:51.232294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:51.232261 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq" Nov 03 22:30:51.232829 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:51.232332 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8" Nov 03 22:30:54.896937 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:54.895134 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:54.896937 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:54.895478 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 03 22:30:55.013598 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:55.012192 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:55.385237 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:55.380182 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" podUID="3b3927af-60a2-4432-99d6-b999a96f315f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:55.517091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:55.513169 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 22:30:55.543244 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:55.543187 5084 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:55.923399 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:55.909507 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 22:30:55.923399 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:55.910280 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:30:55.923399 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:55.911522 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Nov 03 22:30:55.923399 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:55.911864 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:30:56.005994 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:56.004677 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:30:56.014581 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:56.012666 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:56.014581 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:56.013138 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:30:56.067605 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:56.067553 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:30:56.384091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:56.364057 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:30:56.608426 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:56.608386 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:56.609304 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:56.609280 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:30:56.762248 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:56.762153 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:30:57.374597 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:57.374556 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt"] Nov 03 22:30:57.385133 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:57.385097 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:30:57.452575 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:57.452536 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt"] Nov 03 22:30:57.474251 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:57.474218 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-cee4c09d-serving-cert\"" Nov 03 22:30:57.524689 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:57.524121 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-openshift-service-ca\") pod \"maas-api-e2e-ait-cee4c09d-677b986f66-582wt\" (UID: \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:30:57.524689 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:57.524285 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-maas-api-tls\") pod \"maas-api-e2e-ait-cee4c09d-677b986f66-582wt\" (UID: \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:30:57.524689 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:57.524477 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w2l2\" (UniqueName: \"kubernetes.io/projected/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-kube-api-access-8w2l2\") pod \"maas-api-e2e-ait-cee4c09d-677b986f66-582wt\" (UID: \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:30:57.524689 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:57.524521 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-metrics-certs\") pod \"maas-api-e2e-ait-cee4c09d-677b986f66-582wt\" (UID: \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:30:57.647366 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:57.647269 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w2l2\" (UniqueName: \"kubernetes.io/projected/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-kube-api-access-8w2l2\") pod \"maas-api-e2e-ait-cee4c09d-677b986f66-582wt\" (UID: \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:30:57.651328 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:57.651281 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-metrics-certs\") pod \"maas-api-e2e-ait-cee4c09d-677b986f66-582wt\" (UID: \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:30:57.659891 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:57.659841 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-openshift-service-ca\") pod \"maas-api-e2e-ait-cee4c09d-677b986f66-582wt\" (UID: \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:30:57.681908 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:57.675971 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-openshift-service-ca\") pod \"maas-api-e2e-ait-cee4c09d-677b986f66-582wt\" (UID: \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:30:57.682304 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:57.678572 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-maas-api-tls\") pod \"maas-api-e2e-ait-cee4c09d-677b986f66-582wt\" (UID: \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:30:57.682668 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:57.679258 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-serving-cert: secret "maas-api-e2e-ait-cee4c09d-serving-cert" not found Nov 03 22:30:57.682867 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:30:57.682848 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-maas-api-tls podName:4becc34f-f258-4cae-b0fd-4cb837a5b2ed nodeName:}" failed. No retries permitted until 2026-11-03 22:30:58.182820225 +0000 UTC m=+1425.776852182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-maas-api-tls") pod "maas-api-e2e-ait-cee4c09d-677b986f66-582wt" (UID: "4becc34f-f258-4cae-b0fd-4cb837a5b2ed") : secret "maas-api-e2e-ait-cee4c09d-serving-cert" not found Nov 03 22:30:57.684273 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:57.683177 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-metrics-certs\") pod \"maas-api-e2e-ait-cee4c09d-677b986f66-582wt\" (UID: \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:30:57.765614 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:57.765576 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w2l2\" (UniqueName: \"kubernetes.io/projected/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-kube-api-access-8w2l2\") pod \"maas-api-e2e-ait-cee4c09d-677b986f66-582wt\" (UID: \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:30:58.194161 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:58.194110 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-maas-api-tls\") pod \"maas-api-e2e-ait-cee4c09d-677b986f66-582wt\" (UID: \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:30:58.196789 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:58.196761 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-maas-api-tls\") pod \"maas-api-e2e-ait-cee4c09d-677b986f66-582wt\" (UID: \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:30:58.306250 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:58.306213 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:30:58.694782 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:30:58.694749 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4becc34f_f258_4cae_b0fd_4cb837a5b2ed.slice/crio-e7d003722288c134478dee2cf702c168f34895cd3c5d71d35be622cd047c9f6f WatchSource:0}: Error finding container e7d003722288c134478dee2cf702c168f34895cd3c5d71d35be622cd047c9f6f: Status 404 returned error can't find the container with id e7d003722288c134478dee2cf702c168f34895cd3c5d71d35be622cd047c9f6f Nov 03 22:30:58.695280 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:58.695100 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt"] Nov 03 22:30:58.810101 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:58.810019 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" event={"ID":"4becc34f-f258-4cae-b0fd-4cb837a5b2ed","Type":"ContainerStarted","Data":"e7d003722288c134478dee2cf702c168f34895cd3c5d71d35be622cd047c9f6f"} Nov 03 22:30:59.379745 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:59.379696 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s"] Nov 03 22:30:59.395944 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:59.395900 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s" Nov 03 22:30:59.497086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:59.474902 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6462ea50-4efc-40bf-8775-39dd4179bd5e-plugins-config-volume\") pod \"payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s\" (UID: \"6462ea50-4efc-40bf-8775-39dd4179bd5e\") " pod="openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s" Nov 03 22:30:59.497086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:59.474959 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5nkc\" (UniqueName: \"kubernetes.io/projected/6462ea50-4efc-40bf-8775-39dd4179bd5e-kube-api-access-q5nkc\") pod \"payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s\" (UID: \"6462ea50-4efc-40bf-8775-39dd4179bd5e\") " pod="openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s" Nov 03 22:30:59.529132 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:59.498870 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e1c72fb7-dockercfg-x9gxd\"" Nov 03 22:30:59.529132 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:59.523757 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s"] Nov 03 22:30:59.556410 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:59.529956 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e1c72fb7\"" Nov 03 22:30:59.669284 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:59.659570 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6462ea50-4efc-40bf-8775-39dd4179bd5e-plugins-config-volume\") pod \"payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s\" (UID: \"6462ea50-4efc-40bf-8775-39dd4179bd5e\") " pod="openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s" Nov 03 22:30:59.669284 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:59.659625 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5nkc\" (UniqueName: \"kubernetes.io/projected/6462ea50-4efc-40bf-8775-39dd4179bd5e-kube-api-access-q5nkc\") pod \"payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s\" (UID: \"6462ea50-4efc-40bf-8775-39dd4179bd5e\") " pod="openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s" Nov 03 22:30:59.696671 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:59.673447 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6462ea50-4efc-40bf-8775-39dd4179bd5e-plugins-config-volume\") pod \"payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s\" (UID: \"6462ea50-4efc-40bf-8775-39dd4179bd5e\") " pod="openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s" Nov 03 22:30:59.769966 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:30:59.769928 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5nkc\" (UniqueName: \"kubernetes.io/projected/6462ea50-4efc-40bf-8775-39dd4179bd5e-kube-api-access-q5nkc\") pod \"payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s\" (UID: \"6462ea50-4efc-40bf-8775-39dd4179bd5e\") " pod="openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s" Nov 03 22:31:00.015272 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:00.015230 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s" Nov 03 22:31:00.407719 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:00.407674 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s"] Nov 03 22:31:00.417211 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:31:00.416858 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6462ea50_4efc_40bf_8775_39dd4179bd5e.slice/crio-ef6a647ba4f53c788202cbedd201e7c65436a15fe3cee62b1f0d19544e4020ea WatchSource:0}: Error finding container ef6a647ba4f53c788202cbedd201e7c65436a15fe3cee62b1f0d19544e4020ea: Status 404 returned error can't find the container with id ef6a647ba4f53c788202cbedd201e7c65436a15fe3cee62b1f0d19544e4020ea Nov 03 22:31:00.895729 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:00.895690 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" event={"ID":"4becc34f-f258-4cae-b0fd-4cb837a5b2ed","Type":"ContainerStarted","Data":"3e531aa1f16ef0dc6c44dd31b6b3336c27a1f78974f02b85ef708d33071a4c09"} Nov 03 22:31:00.896199 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:00.895870 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:31:00.898988 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:00.898954 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s" event={"ID":"6462ea50-4efc-40bf-8775-39dd4179bd5e","Type":"ContainerStarted","Data":"0c21a6c6bf2a5ef7577233753892d3ab46a6c68b83020e4e5e71d50e9a2ac70a"} Nov 03 22:31:00.899182 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:00.899010 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s" event={"ID":"6462ea50-4efc-40bf-8775-39dd4179bd5e","Type":"ContainerStarted","Data":"ef6a647ba4f53c788202cbedd201e7c65436a15fe3cee62b1f0d19544e4020ea"} Nov 03 22:31:00.899182 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:00.899048 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s" Nov 03 22:31:01.026342 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:01.026282 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" podStartSLOduration=2.839853211 podStartE2EDuration="4.026266197s" totalImagesPullingTime="1.186412986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:01.021358686 +0000 UTC m=+1428.615390660" watchObservedRunningTime="2026-11-03 22:31:01.026266197 +0000 UTC m=+1428.620298170" Nov 03 22:31:01.195144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:01.168642 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s" podStartSLOduration=2.168615993 podStartE2EDuration="2.168615993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:01.153401836 +0000 UTC m=+1428.747433813" watchObservedRunningTime="2026-11-03 22:31:01.168615993 +0000 UTC m=+1428.762647968" Nov 03 22:31:02.633736 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:02.633700 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh"] Nov 03 22:31:02.638416 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:02.638385 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:31:02.659878 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:02.659826 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/705730b8-4c2d-42b2-b9df-005bf8005a15-openshift-service-ca\") pod \"maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh\" (UID: \"705730b8-4c2d-42b2-b9df-005bf8005a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:31:02.660122 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:02.660055 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/705730b8-4c2d-42b2-b9df-005bf8005a15-metrics-certs\") pod \"maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh\" (UID: \"705730b8-4c2d-42b2-b9df-005bf8005a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:31:02.660201 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:02.660162 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whxfc\" (UniqueName: \"kubernetes.io/projected/705730b8-4c2d-42b2-b9df-005bf8005a15-kube-api-access-whxfc\") pod \"maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh\" (UID: \"705730b8-4c2d-42b2-b9df-005bf8005a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:31:02.660343 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:02.660329 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/705730b8-4c2d-42b2-b9df-005bf8005a15-maas-api-tls\") pod \"maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh\" (UID: \"705730b8-4c2d-42b2-b9df-005bf8005a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:31:02.693640 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:02.693581 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh"] Nov 03 22:31:02.737668 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:02.737625 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e6e7831d-serving-cert\"" Nov 03 22:31:02.761803 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:02.761757 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/705730b8-4c2d-42b2-b9df-005bf8005a15-maas-api-tls\") pod \"maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh\" (UID: \"705730b8-4c2d-42b2-b9df-005bf8005a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:31:02.761998 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:02.761832 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/705730b8-4c2d-42b2-b9df-005bf8005a15-openshift-service-ca\") pod \"maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh\" (UID: \"705730b8-4c2d-42b2-b9df-005bf8005a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:31:02.761998 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:02.761916 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/705730b8-4c2d-42b2-b9df-005bf8005a15-metrics-certs\") pod \"maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh\" (UID: \"705730b8-4c2d-42b2-b9df-005bf8005a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:31:02.761998 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:31:02.761951 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-serving-cert: secret "maas-api-e2e-ait-e6e7831d-serving-cert" not found Nov 03 22:31:02.762226 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:02.761959 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whxfc\" (UniqueName: \"kubernetes.io/projected/705730b8-4c2d-42b2-b9df-005bf8005a15-kube-api-access-whxfc\") pod \"maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh\" (UID: \"705730b8-4c2d-42b2-b9df-005bf8005a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:31:02.762226 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:31:02.762036 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/705730b8-4c2d-42b2-b9df-005bf8005a15-maas-api-tls podName:705730b8-4c2d-42b2-b9df-005bf8005a15 nodeName:}" failed. No retries permitted until 2026-11-03 22:31:03.262013878 +0000 UTC m=+1430.856045834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/705730b8-4c2d-42b2-b9df-005bf8005a15-maas-api-tls") pod "maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" (UID: "705730b8-4c2d-42b2-b9df-005bf8005a15") : secret "maas-api-e2e-ait-e6e7831d-serving-cert" not found Nov 03 22:31:02.762782 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:02.762757 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/705730b8-4c2d-42b2-b9df-005bf8005a15-openshift-service-ca\") pod \"maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh\" (UID: \"705730b8-4c2d-42b2-b9df-005bf8005a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:31:02.764683 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:02.764657 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/705730b8-4c2d-42b2-b9df-005bf8005a15-metrics-certs\") pod \"maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh\" (UID: \"705730b8-4c2d-42b2-b9df-005bf8005a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:31:02.852303 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:02.852265 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whxfc\" (UniqueName: \"kubernetes.io/projected/705730b8-4c2d-42b2-b9df-005bf8005a15-kube-api-access-whxfc\") pod \"maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh\" (UID: \"705730b8-4c2d-42b2-b9df-005bf8005a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:31:03.284177 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:03.284122 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/705730b8-4c2d-42b2-b9df-005bf8005a15-maas-api-tls\") pod \"maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh\" (UID: \"705730b8-4c2d-42b2-b9df-005bf8005a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:31:03.286872 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:03.286834 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/705730b8-4c2d-42b2-b9df-005bf8005a15-maas-api-tls\") pod \"maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh\" (UID: \"705730b8-4c2d-42b2-b9df-005bf8005a15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:31:03.557771 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:03.557670 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:31:03.955768 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:31:03.955727 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705730b8_4c2d_42b2_b9df_005bf8005a15.slice/crio-dd2301ac2960bed11bfc9e83f9173211990bbe9a386b7df6616da3bd27938499 WatchSource:0}: Error finding container dd2301ac2960bed11bfc9e83f9173211990bbe9a386b7df6616da3bd27938499: Status 404 returned error can't find the container with id dd2301ac2960bed11bfc9e83f9173211990bbe9a386b7df6616da3bd27938499 Nov 03 22:31:03.956952 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:03.956919 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh"] Nov 03 22:31:03.968783 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:03.968744 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" event={"ID":"705730b8-4c2d-42b2-b9df-005bf8005a15","Type":"ContainerStarted","Data":"dd2301ac2960bed11bfc9e83f9173211990bbe9a386b7df6616da3bd27938499"} Nov 03 22:31:06.020386 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:06.020332 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" event={"ID":"705730b8-4c2d-42b2-b9df-005bf8005a15","Type":"ContainerStarted","Data":"9fc715554b5919f1564b8ed42d40ba66122d936623129eb83fdb8092c0edff2d"} Nov 03 22:31:06.021350 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:06.021324 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:31:06.142540 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:06.142486 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" podStartSLOduration=3.183751575 podStartE2EDuration="4.142468065s" totalImagesPullingTime="958.71649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:06.140499165 +0000 UTC m=+1433.734531137" watchObservedRunningTime="2026-11-03 22:31:06.142468065 +0000 UTC m=+1433.736500038" Nov 03 22:31:06.921463 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:06.921415 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:31:07.817674 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:07.817634 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb"] Nov 03 22:31:07.822786 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:07.822757 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:31:07.875331 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:07.875291 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb"] Nov 03 22:31:07.920130 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:07.920044 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e3937145-serving-cert\"" Nov 03 22:31:07.983582 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:07.983539 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb5m2\" (UniqueName: \"kubernetes.io/projected/23af7f39-b236-441f-af8e-4c8a949f21cb-kube-api-access-wb5m2\") pod \"maas-api-e2e-derive-e3937145-679f5895f5-qf7fb\" (UID: \"23af7f39-b236-441f-af8e-4c8a949f21cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:31:07.983800 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:07.983653 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23af7f39-b236-441f-af8e-4c8a949f21cb-openshift-service-ca\") pod \"maas-api-e2e-derive-e3937145-679f5895f5-qf7fb\" (UID: \"23af7f39-b236-441f-af8e-4c8a949f21cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:31:07.983902 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:07.983879 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23af7f39-b236-441f-af8e-4c8a949f21cb-metrics-certs\") pod \"maas-api-e2e-derive-e3937145-679f5895f5-qf7fb\" (UID: \"23af7f39-b236-441f-af8e-4c8a949f21cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:31:07.983948 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:07.983920 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23af7f39-b236-441f-af8e-4c8a949f21cb-maas-api-tls\") pod \"maas-api-e2e-derive-e3937145-679f5895f5-qf7fb\" (UID: \"23af7f39-b236-441f-af8e-4c8a949f21cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:31:08.088377 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:08.087297 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb5m2\" (UniqueName: \"kubernetes.io/projected/23af7f39-b236-441f-af8e-4c8a949f21cb-kube-api-access-wb5m2\") pod \"maas-api-e2e-derive-e3937145-679f5895f5-qf7fb\" (UID: \"23af7f39-b236-441f-af8e-4c8a949f21cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:31:08.088377 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:08.087372 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23af7f39-b236-441f-af8e-4c8a949f21cb-openshift-service-ca\") pod \"maas-api-e2e-derive-e3937145-679f5895f5-qf7fb\" (UID: \"23af7f39-b236-441f-af8e-4c8a949f21cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:31:08.088377 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:08.087549 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23af7f39-b236-441f-af8e-4c8a949f21cb-metrics-certs\") pod \"maas-api-e2e-derive-e3937145-679f5895f5-qf7fb\" (UID: \"23af7f39-b236-441f-af8e-4c8a949f21cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:31:08.088377 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:08.087580 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23af7f39-b236-441f-af8e-4c8a949f21cb-maas-api-tls\") pod \"maas-api-e2e-derive-e3937145-679f5895f5-qf7fb\" (UID: \"23af7f39-b236-441f-af8e-4c8a949f21cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:31:08.088377 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:31:08.088113 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-serving-cert: secret "maas-api-e2e-derive-e3937145-serving-cert" not found Nov 03 22:31:08.088377 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:31:08.088194 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23af7f39-b236-441f-af8e-4c8a949f21cb-maas-api-tls podName:23af7f39-b236-441f-af8e-4c8a949f21cb nodeName:}" failed. No retries permitted until 2026-11-03 22:31:08.588171616 +0000 UTC m=+1436.182203571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/23af7f39-b236-441f-af8e-4c8a949f21cb-maas-api-tls") pod "maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" (UID: "23af7f39-b236-441f-af8e-4c8a949f21cb") : secret "maas-api-e2e-derive-e3937145-serving-cert" not found Nov 03 22:31:08.100091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:08.088822 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23af7f39-b236-441f-af8e-4c8a949f21cb-openshift-service-ca\") pod \"maas-api-e2e-derive-e3937145-679f5895f5-qf7fb\" (UID: \"23af7f39-b236-441f-af8e-4c8a949f21cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:31:08.131091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:08.128966 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23af7f39-b236-441f-af8e-4c8a949f21cb-metrics-certs\") pod \"maas-api-e2e-derive-e3937145-679f5895f5-qf7fb\" (UID: \"23af7f39-b236-441f-af8e-4c8a949f21cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:31:08.166139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:08.165421 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb5m2\" (UniqueName: \"kubernetes.io/projected/23af7f39-b236-441f-af8e-4c8a949f21cb-kube-api-access-wb5m2\") pod \"maas-api-e2e-derive-e3937145-679f5895f5-qf7fb\" (UID: \"23af7f39-b236-441f-af8e-4c8a949f21cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:31:08.599204 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:08.599158 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23af7f39-b236-441f-af8e-4c8a949f21cb-maas-api-tls\") pod \"maas-api-e2e-derive-e3937145-679f5895f5-qf7fb\" (UID: \"23af7f39-b236-441f-af8e-4c8a949f21cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:31:08.603490 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:08.603453 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23af7f39-b236-441f-af8e-4c8a949f21cb-maas-api-tls\") pod \"maas-api-e2e-derive-e3937145-679f5895f5-qf7fb\" (UID: \"23af7f39-b236-441f-af8e-4c8a949f21cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:31:08.742727 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:08.742685 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:31:09.138230 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:31:09.138199 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23af7f39_b236_441f_af8e_4c8a949f21cb.slice/crio-3df98fda2303715ee5f414ea9a3c84873cb1b6f6f337c32dd73dd33737b7627e WatchSource:0}: Error finding container 3df98fda2303715ee5f414ea9a3c84873cb1b6f6f337c32dd73dd33737b7627e: Status 404 returned error can't find the container with id 3df98fda2303715ee5f414ea9a3c84873cb1b6f6f337c32dd73dd33737b7627e Nov 03 22:31:09.139595 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:09.139560 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb"] Nov 03 22:31:10.057742 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:10.057702 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" event={"ID":"23af7f39-b236-441f-af8e-4c8a949f21cb","Type":"ContainerStarted","Data":"3df98fda2303715ee5f414ea9a3c84873cb1b6f6f337c32dd73dd33737b7627e"} Nov 03 22:31:11.068483 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:11.068411 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" event={"ID":"23af7f39-b236-441f-af8e-4c8a949f21cb","Type":"ContainerStarted","Data":"57c99ba79d8f80365e0005b9ef3d955d5218389f926b55789b500c9a7812acd9"} Nov 03 22:31:11.068892 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:11.068522 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:31:11.189961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:11.189900 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" podStartSLOduration=3.348083187 podStartE2EDuration="4.189883622s" totalImagesPullingTime="841.800435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:11.187392042 +0000 UTC m=+1438.781424017" watchObservedRunningTime="2026-11-03 22:31:11.189883622 +0000 UTC m=+1438.783915595" Nov 03 22:31:11.918601 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:11.918569 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s" Nov 03 22:31:12.035700 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:31:12.035658 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:31:12.035915 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:12.035878 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:31:17.086232 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:17.086196 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:31:17.311405 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:17.311362 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr"] Nov 03 22:31:17.317701 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:17.317669 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:31:17.375748 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:17.375396 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-59a07a45-serving-cert\"" Nov 03 22:31:17.390160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:17.390119 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr"] Nov 03 22:31:17.404719 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:17.404680 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a2a8e8f-3773-45fa-87dd-23445af497d7-metrics-certs\") pod \"maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr\" (UID: \"7a2a8e8f-3773-45fa-87dd-23445af497d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:31:17.404934 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:17.404767 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2a8e8f-3773-45fa-87dd-23445af497d7-openshift-service-ca\") pod \"maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr\" (UID: \"7a2a8e8f-3773-45fa-87dd-23445af497d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:31:17.405010 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:17.404998 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a2a8e8f-3773-45fa-87dd-23445af497d7-maas-api-tls\") pod \"maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr\" (UID: \"7a2a8e8f-3773-45fa-87dd-23445af497d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:31:17.405087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:17.405043 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h4hj\" (UniqueName: \"kubernetes.io/projected/7a2a8e8f-3773-45fa-87dd-23445af497d7-kube-api-access-7h4hj\") pod \"maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr\" (UID: \"7a2a8e8f-3773-45fa-87dd-23445af497d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:31:17.506572 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:17.506519 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a2a8e8f-3773-45fa-87dd-23445af497d7-maas-api-tls\") pod \"maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr\" (UID: \"7a2a8e8f-3773-45fa-87dd-23445af497d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:31:17.506795 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:17.506616 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h4hj\" (UniqueName: \"kubernetes.io/projected/7a2a8e8f-3773-45fa-87dd-23445af497d7-kube-api-access-7h4hj\") pod \"maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr\" (UID: \"7a2a8e8f-3773-45fa-87dd-23445af497d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:31:17.506795 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:31:17.506702 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-serving-cert: secret "maas-api-e2e-migrate-59a07a45-serving-cert" not found Nov 03 22:31:17.506795 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:17.506756 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a2a8e8f-3773-45fa-87dd-23445af497d7-metrics-certs\") pod \"maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr\" (UID: \"7a2a8e8f-3773-45fa-87dd-23445af497d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:31:17.506795 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:31:17.506784 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a2a8e8f-3773-45fa-87dd-23445af497d7-maas-api-tls podName:7a2a8e8f-3773-45fa-87dd-23445af497d7 nodeName:}" failed. No retries permitted until 2026-11-03 22:31:18.006763709 +0000 UTC m=+1445.600795677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a2a8e8f-3773-45fa-87dd-23445af497d7-maas-api-tls") pod "maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" (UID: "7a2a8e8f-3773-45fa-87dd-23445af497d7") : secret "maas-api-e2e-migrate-59a07a45-serving-cert" not found Nov 03 22:31:17.507017 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:17.506824 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2a8e8f-3773-45fa-87dd-23445af497d7-openshift-service-ca\") pod \"maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr\" (UID: \"7a2a8e8f-3773-45fa-87dd-23445af497d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:31:17.507736 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:17.507710 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2a8e8f-3773-45fa-87dd-23445af497d7-openshift-service-ca\") pod \"maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr\" (UID: \"7a2a8e8f-3773-45fa-87dd-23445af497d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:31:17.511947 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:17.511914 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a2a8e8f-3773-45fa-87dd-23445af497d7-metrics-certs\") pod \"maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr\" (UID: \"7a2a8e8f-3773-45fa-87dd-23445af497d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:31:17.567429 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:17.567389 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h4hj\" (UniqueName: \"kubernetes.io/projected/7a2a8e8f-3773-45fa-87dd-23445af497d7-kube-api-access-7h4hj\") pod \"maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr\" (UID: \"7a2a8e8f-3773-45fa-87dd-23445af497d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:31:18.038332 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:18.038285 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a2a8e8f-3773-45fa-87dd-23445af497d7-maas-api-tls\") pod \"maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr\" (UID: \"7a2a8e8f-3773-45fa-87dd-23445af497d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:31:18.040875 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:18.040844 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a2a8e8f-3773-45fa-87dd-23445af497d7-maas-api-tls\") pod \"maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr\" (UID: \"7a2a8e8f-3773-45fa-87dd-23445af497d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:31:18.233440 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:18.233398 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:31:18.628654 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:31:18.628623 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2a8e8f_3773_45fa_87dd_23445af497d7.slice/crio-3f2044678409d0da39327bfb03d88a6d8fe5b0ebafb6468440afdf9ba3e65138 WatchSource:0}: Error finding container 3f2044678409d0da39327bfb03d88a6d8fe5b0ebafb6468440afdf9ba3e65138: Status 404 returned error can't find the container with id 3f2044678409d0da39327bfb03d88a6d8fe5b0ebafb6468440afdf9ba3e65138 Nov 03 22:31:18.631508 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:18.631485 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr"] Nov 03 22:31:19.140581 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:19.140543 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" event={"ID":"7a2a8e8f-3773-45fa-87dd-23445af497d7","Type":"ContainerStarted","Data":"3f2044678409d0da39327bfb03d88a6d8fe5b0ebafb6468440afdf9ba3e65138"} Nov 03 22:31:20.150124 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:20.150044 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" event={"ID":"7a2a8e8f-3773-45fa-87dd-23445af497d7","Type":"ContainerStarted","Data":"d47356d6cad2921b6f1756d1177587a7d31a7b55fe69e4053119ebe64090be49"} Nov 03 22:31:20.150639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:20.150320 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:31:20.275449 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:20.275375 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" podStartSLOduration=2.291869615 podStartE2EDuration="3.275357997s" totalImagesPullingTime="983.488382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:20.270839275 +0000 UTC m=+1447.864871253" watchObservedRunningTime="2026-11-03 22:31:20.275357997 +0000 UTC m=+1447.869389970" Nov 03 22:31:23.260974 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.260930 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982"] Nov 03 22:31:23.267303 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.267260 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.361817 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.361769 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.362051 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.361840 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.362051 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.361993 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.362155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.362050 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.362195 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.362170 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nd4j\" (UniqueName: \"kubernetes.io/projected/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-kube-api-access-2nd4j\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.362238 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.362210 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.362347 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.362331 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.362409 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.362394 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.362477 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.362462 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.403768 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.403733 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982"] Nov 03 22:31:23.464596 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.464534 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.465056 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.465029 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.465345 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.465324 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.465708 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.465671 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.465879 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.465854 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-credential-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.465962 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.465035 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-workload-certs\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.466123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.466096 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-workload-socket\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.466321 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.466299 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.466405 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.466347 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.466538 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.466429 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.466538 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.466466 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.466651 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.466583 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nd4j\" (UniqueName: \"kubernetes.io/projected/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-kube-api-access-2nd4j\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.467809 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.467756 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istiod-ca-cert\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.468052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.468029 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-data\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.470114 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.469293 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-envoy\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.470114 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.469832 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-podinfo\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.527590 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.527552 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-token\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.527790 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.527768 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nd4j\" (UniqueName: \"kubernetes.io/projected/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-kube-api-access-2nd4j\") pod \"e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:23.586501 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:23.586444 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:24.038014 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:24.037968 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982"] Nov 03 22:31:24.038920 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:31:24.038870 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93bd9f01_32b4_42ae_9ef8_f0bef05b18b2.slice/crio-7244b61f63ecb2da7c7c06dda52de420fdacc8711a91758de2e2f21d02d83fd3 WatchSource:0}: Error finding container 7244b61f63ecb2da7c7c06dda52de420fdacc8711a91758de2e2f21d02d83fd3: Status 404 returned error can't find the container with id 7244b61f63ecb2da7c7c06dda52de420fdacc8711a91758de2e2f21d02d83fd3 Nov 03 22:31:24.041952 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:24.041906 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:31:24.042072 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:24.041989 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:31:24.042072 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:24.042022 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:31:24.190675 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:24.190635 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" event={"ID":"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2","Type":"ContainerStarted","Data":"3d8fafceef96b31382c67eeb1086afd63d77914c90f6710de3d67a4774cfdadd"} Nov 03 22:31:24.190886 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:24.190701 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" event={"ID":"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2","Type":"ContainerStarted","Data":"7244b61f63ecb2da7c7c06dda52de420fdacc8711a91758de2e2f21d02d83fd3"} Nov 03 22:31:24.316330 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:24.316216 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" podStartSLOduration=1.316201743 podStartE2EDuration="1.316201743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:24.313000848 +0000 UTC m=+1451.907032823" watchObservedRunningTime="2026-11-03 22:31:24.316201743 +0000 UTC m=+1451.910233717" Nov 03 22:31:24.587281 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:24.587188 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:25.598162 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:25.598122 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 22:31:25.598781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:25.598742 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" podUID="93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 22:31:26.223147 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:26.223107 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:31:26.588091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:26.587595 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 22:31:26.588091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:26.587692 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" podUID="93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 22:31:27.722085 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:27.647368 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 22:31:27.722085 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:27.647445 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" podUID="93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 22:31:28.590523 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:28.590479 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 22:31:28.590846 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:28.590809 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" podUID="93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 22:31:29.611086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:29.610950 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 22:31:29.611086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:29.611016 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" podUID="93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 22:31:30.600120 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:30.599545 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 22:31:30.600120 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:30.599615 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" podUID="93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 22:31:31.591088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:31.590520 5084 patch_prober.go:29] interesting pod/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 22:31:31.591088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:31.590593 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" podUID="93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 22:31:33.103537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:33.103491 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:33.104413 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:33.104385 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:33.106013 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:33.105989 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:31:37.955899 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:37.955806 5084 generic.go:356] "Generic (PLEG): container finished" podID="15696e53-b286-40ae-b2f2-fa6234b6426b" containerID="e8f718ea0253d6a780a8f56d489a04e9a98b07a7dd45c8306057ab353a02dfde" exitCode=0 Nov 03 22:31:37.956384 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:37.955889 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" event={"ID":"15696e53-b286-40ae-b2f2-fa6234b6426b","Type":"ContainerDied","Data":"e8f718ea0253d6a780a8f56d489a04e9a98b07a7dd45c8306057ab353a02dfde"} Nov 03 22:31:37.956519 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:37.956494 5084 scope.go:122] "RemoveContainer" containerID="e8f718ea0253d6a780a8f56d489a04e9a98b07a7dd45c8306057ab353a02dfde" Nov 03 22:31:37.958686 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:37.958645 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:31:37.958785 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:37.958751 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:31:37.958838 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:37.958821 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:31:39.004311 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:39.004273 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" event={"ID":"15696e53-b286-40ae-b2f2-fa6234b6426b","Type":"ContainerStarted","Data":"54c5eef7445896f46907fd3a58d759dec0088d8390afb8914eedc291ebd58639"} Nov 03 22:31:39.130824 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:39.130780 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:31:39.131928 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:39.131895 5084 patch_prober.go:29] interesting pod/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:31:39.132083 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:39.131986 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" podUID="15696e53-b286-40ae-b2f2-fa6234b6426b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 22:31:39.634871 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:39.634827 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:31:40.131241 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:40.131197 5084 patch_prober.go:29] interesting pod/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:31:40.131666 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:40.131264 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" podUID="15696e53-b286-40ae-b2f2-fa6234b6426b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 22:31:41.130670 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:41.130625 5084 patch_prober.go:29] interesting pod/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:31:41.130947 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:41.130697 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" podUID="15696e53-b286-40ae-b2f2-fa6234b6426b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 22:31:42.028544 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:31:42.028500 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:31:42.130835 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:42.130795 5084 patch_prober.go:29] interesting pod/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:31:42.131027 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:42.130854 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" podUID="15696e53-b286-40ae-b2f2-fa6234b6426b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 22:31:43.130714 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:43.130678 5084 patch_prober.go:29] interesting pod/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:31:43.131168 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:43.130735 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" podUID="15696e53-b286-40ae-b2f2-fa6234b6426b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 22:31:44.130744 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:44.130703 5084 patch_prober.go:29] interesting pod/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 22:31:44.131195 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:44.130780 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" podUID="15696e53-b286-40ae-b2f2-fa6234b6426b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 22:31:45.135129 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:45.135075 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:31:45.136302 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:45.136276 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:31:50.521805 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:50.521758 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw"] Nov 03 22:31:50.527906 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:50.527873 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:31:50.578633 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:50.578589 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw"] Nov 03 22:31:50.624271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:50.624231 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e1c72fb7-serving-cert\"" Nov 03 22:31:50.626704 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:50.626664 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm8pk\" (UniqueName: \"kubernetes.io/projected/e5034590-d620-46de-92ee-22cf08dd39be-kube-api-access-xm8pk\") pod \"maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw\" (UID: \"e5034590-d620-46de-92ee-22cf08dd39be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:31:50.626888 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:50.626844 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5034590-d620-46de-92ee-22cf08dd39be-metrics-certs\") pod \"maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw\" (UID: \"e5034590-d620-46de-92ee-22cf08dd39be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:31:50.626977 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:50.626964 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5034590-d620-46de-92ee-22cf08dd39be-maas-api-tls\") pod \"maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw\" (UID: \"e5034590-d620-46de-92ee-22cf08dd39be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:31:50.627049 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:50.627028 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5034590-d620-46de-92ee-22cf08dd39be-openshift-service-ca\") pod \"maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw\" (UID: \"e5034590-d620-46de-92ee-22cf08dd39be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:31:50.728196 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:50.728160 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5034590-d620-46de-92ee-22cf08dd39be-metrics-certs\") pod \"maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw\" (UID: \"e5034590-d620-46de-92ee-22cf08dd39be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:31:50.728420 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:50.728264 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5034590-d620-46de-92ee-22cf08dd39be-maas-api-tls\") pod \"maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw\" (UID: \"e5034590-d620-46de-92ee-22cf08dd39be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:31:50.728420 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:50.728289 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5034590-d620-46de-92ee-22cf08dd39be-openshift-service-ca\") pod \"maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw\" (UID: \"e5034590-d620-46de-92ee-22cf08dd39be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:31:50.728420 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:31:50.728349 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-serving-cert: secret "maas-api-e2e-mt-e1c72fb7-serving-cert" not found Nov 03 22:31:50.728577 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:31:50.728421 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5034590-d620-46de-92ee-22cf08dd39be-maas-api-tls podName:e5034590-d620-46de-92ee-22cf08dd39be nodeName:}" failed. No retries permitted until 2026-11-03 22:31:51.228403784 +0000 UTC m=+1478.822435738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e5034590-d620-46de-92ee-22cf08dd39be-maas-api-tls") pod "maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" (UID: "e5034590-d620-46de-92ee-22cf08dd39be") : secret "maas-api-e2e-mt-e1c72fb7-serving-cert" not found Nov 03 22:31:50.728577 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:50.728502 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm8pk\" (UniqueName: \"kubernetes.io/projected/e5034590-d620-46de-92ee-22cf08dd39be-kube-api-access-xm8pk\") pod \"maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw\" (UID: \"e5034590-d620-46de-92ee-22cf08dd39be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:31:50.729149 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:50.729122 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5034590-d620-46de-92ee-22cf08dd39be-openshift-service-ca\") pod \"maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw\" (UID: \"e5034590-d620-46de-92ee-22cf08dd39be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:31:50.730908 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:50.730884 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5034590-d620-46de-92ee-22cf08dd39be-metrics-certs\") pod \"maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw\" (UID: \"e5034590-d620-46de-92ee-22cf08dd39be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:31:50.793856 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:50.792603 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm8pk\" (UniqueName: \"kubernetes.io/projected/e5034590-d620-46de-92ee-22cf08dd39be-kube-api-access-xm8pk\") pod \"maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw\" (UID: \"e5034590-d620-46de-92ee-22cf08dd39be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:31:51.253501 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:51.253407 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5034590-d620-46de-92ee-22cf08dd39be-maas-api-tls\") pod \"maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw\" (UID: \"e5034590-d620-46de-92ee-22cf08dd39be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:31:51.256160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:51.256129 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5034590-d620-46de-92ee-22cf08dd39be-maas-api-tls\") pod \"maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw\" (UID: \"e5034590-d620-46de-92ee-22cf08dd39be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:31:51.444176 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:51.444135 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:31:51.829643 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:51.829610 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw"] Nov 03 22:31:51.830135 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:31:51.829873 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5034590_d620_46de_92ee_22cf08dd39be.slice/crio-0187afd19c6abf8a1c458793cb9d43bc9b4dccbd370ad9ed763a8bdab7082942 WatchSource:0}: Error finding container 0187afd19c6abf8a1c458793cb9d43bc9b4dccbd370ad9ed763a8bdab7082942: Status 404 returned error can't find the container with id 0187afd19c6abf8a1c458793cb9d43bc9b4dccbd370ad9ed763a8bdab7082942 Nov 03 22:31:52.134396 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:52.134285 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" event={"ID":"e5034590-d620-46de-92ee-22cf08dd39be","Type":"ContainerStarted","Data":"0187afd19c6abf8a1c458793cb9d43bc9b4dccbd370ad9ed763a8bdab7082942"} Nov 03 22:31:53.145029 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:53.144928 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" event={"ID":"e5034590-d620-46de-92ee-22cf08dd39be","Type":"ContainerStarted","Data":"5a75ccb86dd938e49281965a2020fd4b1cf74e54de1b91c9b5c693b961b37b7c"} Nov 03 22:31:53.145540 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:53.145037 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:31:53.266159 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:53.266101 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" podStartSLOduration=2.2878805509999998 podStartE2EDuration="3.266079799s" totalImagesPullingTime="978.199248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:53.261995907 +0000 UTC m=+1480.856027880" watchObservedRunningTime="2026-11-03 22:31:53.266079799 +0000 UTC m=+1480.860111770" Nov 03 22:31:59.160112 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:31:59.160058 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:32:12.028320 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:32:12.028276 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:32:12.849451 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:12.849421 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:32:12.871092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:12.868218 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:32:20.589595 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:20.589556 5084 generic.go:356] "Generic (PLEG): container finished" podID="3afecb8a-983a-4938-a155-c9561b7c1144" containerID="f444bbf9c1bcce1f1e97471f788b3d795fdbe633222732dc0675131462755ca2" exitCode=0 Nov 03 22:32:20.590046 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:20.589744 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" event={"ID":"3afecb8a-983a-4938-a155-c9561b7c1144","Type":"ContainerDied","Data":"f444bbf9c1bcce1f1e97471f788b3d795fdbe633222732dc0675131462755ca2"} Nov 03 22:32:20.590252 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:20.590232 5084 scope.go:122] "RemoveContainer" containerID="f444bbf9c1bcce1f1e97471f788b3d795fdbe633222732dc0675131462755ca2" Nov 03 22:32:20.592403 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:20.592360 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:32:20.592485 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:20.592465 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:32:20.592531 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:20.592516 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:32:21.602017 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:21.601970 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" event={"ID":"3afecb8a-983a-4938-a155-c9561b7c1144","Type":"ContainerStarted","Data":"7609f13f6d28f1ccb379d04a7b0cdebc265bb27f3b7a6fab34a18c817e938dd8"} Nov 03 22:32:22.596618 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:22.596567 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:32:22.598363 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:22.598331 5084 patch_prober.go:29] interesting pod/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:32:22.598494 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:22.598398 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" podUID="3afecb8a-983a-4938-a155-c9561b7c1144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 22:32:23.596678 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:23.596636 5084 patch_prober.go:29] interesting pod/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:32:23.597206 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:23.596724 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" podUID="3afecb8a-983a-4938-a155-c9561b7c1144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 22:32:23.618702 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:23.618672 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:32:24.596009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:24.595969 5084 patch_prober.go:29] interesting pod/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:32:24.596233 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:24.596029 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" podUID="3afecb8a-983a-4938-a155-c9561b7c1144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 22:32:25.596010 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:25.595963 5084 patch_prober.go:29] interesting pod/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:32:25.596477 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:25.596033 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" podUID="3afecb8a-983a-4938-a155-c9561b7c1144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 22:32:26.596736 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:26.596696 5084 patch_prober.go:29] interesting pod/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:32:26.597187 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:26.596756 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" podUID="3afecb8a-983a-4938-a155-c9561b7c1144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 22:32:27.601596 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:27.601562 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:32:27.602759 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:27.602734 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:32:41.052420 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:41.052369 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2"] Nov 03 22:32:41.103878 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:41.103825 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79aa88d1-h6zw2"] Nov 03 22:32:42.025445 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:32:42.025402 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:32:42.629983 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:42.629944 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10378c78-d3be-461e-ad15-ff38ecf8d7ea" path="/var/lib/kubelet/pods/10378c78-d3be-461e-ad15-ff38ecf8d7ea/volumes" Nov 03 22:32:55.394612 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:55.394572 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq"] Nov 03 22:32:55.515351 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:32:55.515316 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8"] Nov 03 22:33:00.394953 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:00.394893 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq" podUID="1ec48958-96f5-4e49-a495-df51c0c5c029" containerName="payload-processing" containerID="cri-o://48a23feeba821c854b473564da943e892b808285208e8783d121980f43e23b4f" gracePeriod=25 Nov 03 22:33:00.515991 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:00.515911 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8" podUID="2eb841e6-f440-4cf4-a36b-4604c2bc92c3" containerName="payload-pre-processing" containerID="cri-o://72eb3968e9ed633b63cf718c2295b4715f159289acb1383f8943a037368530a7" gracePeriod=25 Nov 03 22:33:00.575019 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:00.574979 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2"] Nov 03 22:33:00.713829 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:00.713000 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj"] Nov 03 22:33:00.762220 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:00.762184 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq" Nov 03 22:33:00.833628 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:00.833592 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec48958-96f5-4e49-a495-df51c0c5c029-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec48958-96f5-4e49-a495-df51c0c5c029-plugins-config-volume\") pod \"1ec48958-96f5-4e49-a495-df51c0c5c029\" (UID: \"1ec48958-96f5-4e49-a495-df51c0c5c029\") " Nov 03 22:33:00.834071 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:00.834025 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec48958-96f5-4e49-a495-df51c0c5c029-kube-api-access-fgw8v\" (UniqueName: \"kubernetes.io/projected/1ec48958-96f5-4e49-a495-df51c0c5c029-kube-api-access-fgw8v\") pod \"1ec48958-96f5-4e49-a495-df51c0c5c029\" (UID: \"1ec48958-96f5-4e49-a495-df51c0c5c029\") " Nov 03 22:33:00.834273 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:00.834240 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec48958-96f5-4e49-a495-df51c0c5c029-plugins-config-volume" pod "1ec48958-96f5-4e49-a495-df51c0c5c029" (UID: "1ec48958-96f5-4e49-a495-df51c0c5c029"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:00.835771 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:00.835744 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec48958-96f5-4e49-a495-df51c0c5c029-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:00.837315 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:00.837278 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec48958-96f5-4e49-a495-df51c0c5c029-kube-api-access-fgw8v" pod "1ec48958-96f5-4e49-a495-df51c0c5c029" (UID: "1ec48958-96f5-4e49-a495-df51c0c5c029"). InnerVolumeSpecName "kube-api-access-fgw8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:00.866715 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:00.866673 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8" Nov 03 22:33:00.937077 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:00.936974 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb841e6-f440-4cf4-a36b-4604c2bc92c3-kube-api-access-q5x29\" (UniqueName: \"kubernetes.io/projected/2eb841e6-f440-4cf4-a36b-4604c2bc92c3-kube-api-access-q5x29\") pod \"2eb841e6-f440-4cf4-a36b-4604c2bc92c3\" (UID: \"2eb841e6-f440-4cf4-a36b-4604c2bc92c3\") " Nov 03 22:33:00.937252 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:00.937238 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eb841e6-f440-4cf4-a36b-4604c2bc92c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb841e6-f440-4cf4-a36b-4604c2bc92c3-plugins-config-volume\") pod \"2eb841e6-f440-4cf4-a36b-4604c2bc92c3\" (UID: \"2eb841e6-f440-4cf4-a36b-4604c2bc92c3\") " Nov 03 22:33:00.937686 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:00.937648 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb841e6-f440-4cf4-a36b-4604c2bc92c3-plugins-config-volume" pod "2eb841e6-f440-4cf4-a36b-4604c2bc92c3" (UID: "2eb841e6-f440-4cf4-a36b-4604c2bc92c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:00.938024 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:00.938004 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgw8v\" (UniqueName: \"kubernetes.io/projected/1ec48958-96f5-4e49-a495-df51c0c5c029-kube-api-access-fgw8v\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:00.938133 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:00.938025 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb841e6-f440-4cf4-a36b-4604c2bc92c3-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:00.939250 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:00.939223 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb841e6-f440-4cf4-a36b-4604c2bc92c3-kube-api-access-q5x29" pod "2eb841e6-f440-4cf4-a36b-4604c2bc92c3" (UID: "2eb841e6-f440-4cf4-a36b-4604c2bc92c3"). InnerVolumeSpecName "kube-api-access-q5x29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:01.054086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.040487 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5x29\" (UniqueName: \"kubernetes.io/projected/2eb841e6-f440-4cf4-a36b-4604c2bc92c3-kube-api-access-q5x29\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:01.376087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.373228 5084 generic.go:356] "Generic (PLEG): container finished" podID="1ec48958-96f5-4e49-a495-df51c0c5c029" containerID="48a23feeba821c854b473564da943e892b808285208e8783d121980f43e23b4f" exitCode=0 Nov 03 22:33:01.376087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.373568 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq" event={"ID":"1ec48958-96f5-4e49-a495-df51c0c5c029","Type":"ContainerDied","Data":"48a23feeba821c854b473564da943e892b808285208e8783d121980f43e23b4f"} Nov 03 22:33:01.376087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.373622 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq" event={"ID":"1ec48958-96f5-4e49-a495-df51c0c5c029","Type":"ContainerDied","Data":"e9e44ba0766ec423dd53aa27c2526a2415461449edbb62d157bee0583d3ac3aa"} Nov 03 22:33:01.376087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.373652 5084 scope.go:122] "RemoveContainer" containerID="48a23feeba821c854b473564da943e892b808285208e8783d121980f43e23b4f" Nov 03 22:33:01.376087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.373851 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq" Nov 03 22:33:01.390697 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.390657 5084 generic.go:356] "Generic (PLEG): container finished" podID="2eb841e6-f440-4cf4-a36b-4604c2bc92c3" containerID="72eb3968e9ed633b63cf718c2295b4715f159289acb1383f8943a037368530a7" exitCode=0 Nov 03 22:33:01.390864 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.390720 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8" event={"ID":"2eb841e6-f440-4cf4-a36b-4604c2bc92c3","Type":"ContainerDied","Data":"72eb3968e9ed633b63cf718c2295b4715f159289acb1383f8943a037368530a7"} Nov 03 22:33:01.390864 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.390724 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8" Nov 03 22:33:01.390864 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.390773 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8" event={"ID":"2eb841e6-f440-4cf4-a36b-4604c2bc92c3","Type":"ContainerDied","Data":"13795cdb6052bf623664b1c9c4b774728b8efdee04e6988e1ea9cd2e24aeb66b"} Nov 03 22:33:01.407760 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.407484 5084 scope.go:122] "RemoveContainer" containerID="48a23feeba821c854b473564da943e892b808285208e8783d121980f43e23b4f" Nov 03 22:33:01.408118 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:33:01.407794 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a23feeba821c854b473564da943e892b808285208e8783d121980f43e23b4f\": container with ID starting with 48a23feeba821c854b473564da943e892b808285208e8783d121980f43e23b4f not found: ID does not exist" containerID="48a23feeba821c854b473564da943e892b808285208e8783d121980f43e23b4f" Nov 03 22:33:01.408118 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.407826 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a23feeba821c854b473564da943e892b808285208e8783d121980f43e23b4f"} err="failed to get container status \"48a23feeba821c854b473564da943e892b808285208e8783d121980f43e23b4f\": rpc error: code = NotFound desc = could not find container \"48a23feeba821c854b473564da943e892b808285208e8783d121980f43e23b4f\": container with ID starting with 48a23feeba821c854b473564da943e892b808285208e8783d121980f43e23b4f not found: ID does not exist" Nov 03 22:33:01.408118 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.407845 5084 scope.go:122] "RemoveContainer" containerID="72eb3968e9ed633b63cf718c2295b4715f159289acb1383f8943a037368530a7" Nov 03 22:33:01.428741 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.428711 5084 scope.go:122] "RemoveContainer" containerID="72eb3968e9ed633b63cf718c2295b4715f159289acb1383f8943a037368530a7" Nov 03 22:33:01.429115 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:33:01.429086 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72eb3968e9ed633b63cf718c2295b4715f159289acb1383f8943a037368530a7\": container with ID starting with 72eb3968e9ed633b63cf718c2295b4715f159289acb1383f8943a037368530a7 not found: ID does not exist" containerID="72eb3968e9ed633b63cf718c2295b4715f159289acb1383f8943a037368530a7" Nov 03 22:33:01.429199 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.429130 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72eb3968e9ed633b63cf718c2295b4715f159289acb1383f8943a037368530a7"} err="failed to get container status \"72eb3968e9ed633b63cf718c2295b4715f159289acb1383f8943a037368530a7\": rpc error: code = NotFound desc = could not find container \"72eb3968e9ed633b63cf718c2295b4715f159289acb1383f8943a037368530a7\": container with ID starting with 72eb3968e9ed633b63cf718c2295b4715f159289acb1383f8943a037368530a7 not found: ID does not exist" Nov 03 22:33:01.558526 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.558486 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq"] Nov 03 22:33:01.566678 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.566640 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e3937145-7f4878969d-wm6lq"] Nov 03 22:33:01.673102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.672980 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8"] Nov 03 22:33:01.678748 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:01.678707 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3937145-fc6d6855-n76c8"] Nov 03 22:33:02.080693 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.080647 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt"] Nov 03 22:33:02.081057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.081002 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" podUID="4becc34f-f258-4cae-b0fd-4cb837a5b2ed" containerName="maas-api" containerID="cri-o://3e531aa1f16ef0dc6c44dd31b6b3336c27a1f78974f02b85ef708d33071a4c09" gracePeriod=30 Nov 03 22:33:02.403963 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.403932 5084 generic.go:356] "Generic (PLEG): container finished" podID="4becc34f-f258-4cae-b0fd-4cb837a5b2ed" containerID="3e531aa1f16ef0dc6c44dd31b6b3336c27a1f78974f02b85ef708d33071a4c09" exitCode=0 Nov 03 22:33:02.404181 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.404003 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" event={"ID":"4becc34f-f258-4cae-b0fd-4cb837a5b2ed","Type":"ContainerDied","Data":"3e531aa1f16ef0dc6c44dd31b6b3336c27a1f78974f02b85ef708d33071a4c09"} Nov 03 22:33:02.404181 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.404047 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" event={"ID":"4becc34f-f258-4cae-b0fd-4cb837a5b2ed","Type":"ContainerDied","Data":"e7d003722288c134478dee2cf702c168f34895cd3c5d71d35be622cd047c9f6f"} Nov 03 22:33:02.404181 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.404107 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7d003722288c134478dee2cf702c168f34895cd3c5d71d35be622cd047c9f6f" Nov 03 22:33:02.410527 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.410490 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:33:02.493275 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.493234 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj"] Nov 03 22:33:02.512824 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.512781 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-kube-api-access-8w2l2\" (UniqueName: \"kubernetes.io/projected/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-kube-api-access-8w2l2\") pod \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\" (UID: \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\") " Nov 03 22:33:02.513004 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.512850 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-maas-api-tls\") pod \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\" (UID: \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\") " Nov 03 22:33:02.513004 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.512954 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-metrics-certs\") pod \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\" (UID: \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\") " Nov 03 22:33:02.513126 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.513015 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-openshift-service-ca\") pod \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\" (UID: \"4becc34f-f258-4cae-b0fd-4cb837a5b2ed\") " Nov 03 22:33:02.513460 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.513429 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-openshift-service-ca" pod "4becc34f-f258-4cae-b0fd-4cb837a5b2ed" (UID: "4becc34f-f258-4cae-b0fd-4cb837a5b2ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:02.513652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.513595 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:02.515554 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.515522 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-metrics-certs" pod "4becc34f-f258-4cae-b0fd-4cb837a5b2ed" (UID: "4becc34f-f258-4cae-b0fd-4cb837a5b2ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:33:02.515761 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.515738 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-kube-api-access-8w2l2" pod "4becc34f-f258-4cae-b0fd-4cb837a5b2ed" (UID: "4becc34f-f258-4cae-b0fd-4cb837a5b2ed"). InnerVolumeSpecName "kube-api-access-8w2l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:02.515831 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.515767 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-maas-api-tls" pod "4becc34f-f258-4cae-b0fd-4cb837a5b2ed" (UID: "4becc34f-f258-4cae-b0fd-4cb837a5b2ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:33:02.550463 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.550417 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff"] Nov 03 22:33:02.615111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.614957 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w2l2\" (UniqueName: \"kubernetes.io/projected/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-kube-api-access-8w2l2\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:02.615111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.614995 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:02.615111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.615005 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4becc34f-f258-4cae-b0fd-4cb837a5b2ed-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:02.634293 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.634245 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec48958-96f5-4e49-a495-df51c0c5c029" path="/var/lib/kubelet/pods/1ec48958-96f5-4e49-a495-df51c0c5c029/volumes" Nov 03 22:33:02.635052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:02.635023 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eb841e6-f440-4cf4-a36b-4604c2bc92c3" path="/var/lib/kubelet/pods/2eb841e6-f440-4cf4-a36b-4604c2bc92c3/volumes" Nov 03 22:33:03.213183 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:03.213136 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh"] Nov 03 22:33:03.233185 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:03.233129 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh"] Nov 03 22:33:03.233889 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:03.233850 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" podUID="705730b8-4c2d-42b2-b9df-005bf8005a15" containerName="maas-api" containerID="cri-o://9fc715554b5919f1564b8ed42d40ba66122d936623129eb83fdb8092c0edff2d" gracePeriod=30 Nov 03 22:33:03.569539 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:03.532694 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt" Nov 03 22:33:03.609672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:03.609130 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb"] Nov 03 22:33:03.732118 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:03.725347 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt"] Nov 03 22:33:03.735121 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:03.733183 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cee4c09d-677b986f66-582wt"] Nov 03 22:33:04.643091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.637652 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:33:04.656152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.650339 5084 generic.go:356] "Generic (PLEG): container finished" podID="705730b8-4c2d-42b2-b9df-005bf8005a15" containerID="9fc715554b5919f1564b8ed42d40ba66122d936623129eb83fdb8092c0edff2d" exitCode=0 Nov 03 22:33:04.656152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.650484 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" Nov 03 22:33:04.678087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.663809 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4becc34f-f258-4cae-b0fd-4cb837a5b2ed" path="/var/lib/kubelet/pods/4becc34f-f258-4cae-b0fd-4cb837a5b2ed/volumes" Nov 03 22:33:04.678087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.672476 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" event={"ID":"705730b8-4c2d-42b2-b9df-005bf8005a15","Type":"ContainerDied","Data":"9fc715554b5919f1564b8ed42d40ba66122d936623129eb83fdb8092c0edff2d"} Nov 03 22:33:04.678087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.672543 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh" event={"ID":"705730b8-4c2d-42b2-b9df-005bf8005a15","Type":"ContainerDied","Data":"dd2301ac2960bed11bfc9e83f9173211990bbe9a386b7df6616da3bd27938499"} Nov 03 22:33:04.678087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.672595 5084 scope.go:122] "RemoveContainer" containerID="9fc715554b5919f1564b8ed42d40ba66122d936623129eb83fdb8092c0edff2d" Nov 03 22:33:04.701087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.685532 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb"] Nov 03 22:33:04.701087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.685766 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" podUID="23af7f39-b236-441f-af8e-4c8a949f21cb" containerName="maas-api" containerID="cri-o://57c99ba79d8f80365e0005b9ef3d955d5218389f926b55789b500c9a7812acd9" gracePeriod=30 Nov 03 22:33:04.745509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.744695 5084 scope.go:122] "RemoveContainer" containerID="9fc715554b5919f1564b8ed42d40ba66122d936623129eb83fdb8092c0edff2d" Nov 03 22:33:04.745699 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:33:04.745634 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc715554b5919f1564b8ed42d40ba66122d936623129eb83fdb8092c0edff2d\": container with ID starting with 9fc715554b5919f1564b8ed42d40ba66122d936623129eb83fdb8092c0edff2d not found: ID does not exist" containerID="9fc715554b5919f1564b8ed42d40ba66122d936623129eb83fdb8092c0edff2d" Nov 03 22:33:04.745783 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.745714 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc715554b5919f1564b8ed42d40ba66122d936623129eb83fdb8092c0edff2d"} err="failed to get container status \"9fc715554b5919f1564b8ed42d40ba66122d936623129eb83fdb8092c0edff2d\": rpc error: code = NotFound desc = could not find container \"9fc715554b5919f1564b8ed42d40ba66122d936623129eb83fdb8092c0edff2d\": container with ID starting with 9fc715554b5919f1564b8ed42d40ba66122d936623129eb83fdb8092c0edff2d not found: ID does not exist" Nov 03 22:33:04.789127 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.787860 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/705730b8-4c2d-42b2-b9df-005bf8005a15-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/705730b8-4c2d-42b2-b9df-005bf8005a15-openshift-service-ca\") pod \"705730b8-4c2d-42b2-b9df-005bf8005a15\" (UID: \"705730b8-4c2d-42b2-b9df-005bf8005a15\") " Nov 03 22:33:04.789127 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.788050 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/705730b8-4c2d-42b2-b9df-005bf8005a15-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/705730b8-4c2d-42b2-b9df-005bf8005a15-maas-api-tls\") pod \"705730b8-4c2d-42b2-b9df-005bf8005a15\" (UID: \"705730b8-4c2d-42b2-b9df-005bf8005a15\") " Nov 03 22:33:04.789127 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.788182 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705730b8-4c2d-42b2-b9df-005bf8005a15-kube-api-access-whxfc\" (UniqueName: \"kubernetes.io/projected/705730b8-4c2d-42b2-b9df-005bf8005a15-kube-api-access-whxfc\") pod \"705730b8-4c2d-42b2-b9df-005bf8005a15\" (UID: \"705730b8-4c2d-42b2-b9df-005bf8005a15\") " Nov 03 22:33:04.789127 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.788256 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/705730b8-4c2d-42b2-b9df-005bf8005a15-metrics-certs\" (UniqueName: \"kubernetes.io/secret/705730b8-4c2d-42b2-b9df-005bf8005a15-metrics-certs\") pod \"705730b8-4c2d-42b2-b9df-005bf8005a15\" (UID: \"705730b8-4c2d-42b2-b9df-005bf8005a15\") " Nov 03 22:33:04.807840 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.802197 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/705730b8-4c2d-42b2-b9df-005bf8005a15-metrics-certs" pod "705730b8-4c2d-42b2-b9df-005bf8005a15" (UID: "705730b8-4c2d-42b2-b9df-005bf8005a15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:33:04.812109 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.811291 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/705730b8-4c2d-42b2-b9df-005bf8005a15-maas-api-tls" pod "705730b8-4c2d-42b2-b9df-005bf8005a15" (UID: "705730b8-4c2d-42b2-b9df-005bf8005a15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:33:04.813090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.812594 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705730b8-4c2d-42b2-b9df-005bf8005a15-openshift-service-ca" pod "705730b8-4c2d-42b2-b9df-005bf8005a15" (UID: "705730b8-4c2d-42b2-b9df-005bf8005a15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:04.826150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.826102 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705730b8-4c2d-42b2-b9df-005bf8005a15-kube-api-access-whxfc" pod "705730b8-4c2d-42b2-b9df-005bf8005a15" (UID: "705730b8-4c2d-42b2-b9df-005bf8005a15"). InnerVolumeSpecName "kube-api-access-whxfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:04.897318 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.897277 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whxfc\" (UniqueName: \"kubernetes.io/projected/705730b8-4c2d-42b2-b9df-005bf8005a15-kube-api-access-whxfc\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:04.897318 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.897317 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/705730b8-4c2d-42b2-b9df-005bf8005a15-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:04.897576 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.897330 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/705730b8-4c2d-42b2-b9df-005bf8005a15-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:04.897576 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:04.897343 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/705730b8-4c2d-42b2-b9df-005bf8005a15-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:05.348665 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.348624 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:33:05.371240 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.371203 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh"] Nov 03 22:33:05.389294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.387933 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e6e7831d-5c5bd969cd-h9phh"] Nov 03 22:33:05.542086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.539576 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23af7f39-b236-441f-af8e-4c8a949f21cb-kube-api-access-wb5m2\" (UniqueName: \"kubernetes.io/projected/23af7f39-b236-441f-af8e-4c8a949f21cb-kube-api-access-wb5m2\") pod \"23af7f39-b236-441f-af8e-4c8a949f21cb\" (UID: \"23af7f39-b236-441f-af8e-4c8a949f21cb\") " Nov 03 22:33:05.542086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.540340 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23af7f39-b236-441f-af8e-4c8a949f21cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23af7f39-b236-441f-af8e-4c8a949f21cb-maas-api-tls\") pod \"23af7f39-b236-441f-af8e-4c8a949f21cb\" (UID: \"23af7f39-b236-441f-af8e-4c8a949f21cb\") " Nov 03 22:33:05.542086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.540696 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23af7f39-b236-441f-af8e-4c8a949f21cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23af7f39-b236-441f-af8e-4c8a949f21cb-openshift-service-ca\") pod \"23af7f39-b236-441f-af8e-4c8a949f21cb\" (UID: \"23af7f39-b236-441f-af8e-4c8a949f21cb\") " Nov 03 22:33:05.542086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.540870 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23af7f39-b236-441f-af8e-4c8a949f21cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/23af7f39-b236-441f-af8e-4c8a949f21cb-metrics-certs\") pod \"23af7f39-b236-441f-af8e-4c8a949f21cb\" (UID: \"23af7f39-b236-441f-af8e-4c8a949f21cb\") " Nov 03 22:33:05.548193 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.548148 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23af7f39-b236-441f-af8e-4c8a949f21cb-openshift-service-ca" pod "23af7f39-b236-441f-af8e-4c8a949f21cb" (UID: "23af7f39-b236-441f-af8e-4c8a949f21cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:05.553963 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.553921 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23af7f39-b236-441f-af8e-4c8a949f21cb-metrics-certs" pod "23af7f39-b236-441f-af8e-4c8a949f21cb" (UID: "23af7f39-b236-441f-af8e-4c8a949f21cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:33:05.567402 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.567333 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23af7f39-b236-441f-af8e-4c8a949f21cb-maas-api-tls" pod "23af7f39-b236-441f-af8e-4c8a949f21cb" (UID: "23af7f39-b236-441f-af8e-4c8a949f21cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:33:05.576195 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.575464 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" podUID="2a5fce82-2002-48a4-ba14-e5e4bf6bfed5" containerName="payload-processing" containerID="cri-o://e61ffeac612291c8923b5bc0f444ff879f9ec99980ef7786565523779afb3427" gracePeriod=25 Nov 03 22:33:05.577306 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.577212 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23af7f39-b236-441f-af8e-4c8a949f21cb-kube-api-access-wb5m2" pod "23af7f39-b236-441f-af8e-4c8a949f21cb" (UID: "23af7f39-b236-441f-af8e-4c8a949f21cb"). InnerVolumeSpecName "kube-api-access-wb5m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:05.643546 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.643502 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb5m2\" (UniqueName: \"kubernetes.io/projected/23af7f39-b236-441f-af8e-4c8a949f21cb-kube-api-access-wb5m2\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:05.643546 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.643539 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23af7f39-b236-441f-af8e-4c8a949f21cb-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:05.643546 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.643549 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23af7f39-b236-441f-af8e-4c8a949f21cb-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:05.643546 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.643558 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23af7f39-b236-441f-af8e-4c8a949f21cb-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:05.662513 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.662474 5084 generic.go:356] "Generic (PLEG): container finished" podID="23af7f39-b236-441f-af8e-4c8a949f21cb" containerID="57c99ba79d8f80365e0005b9ef3d955d5218389f926b55789b500c9a7812acd9" exitCode=0 Nov 03 22:33:05.662699 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.662524 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" Nov 03 22:33:05.662699 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.662560 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" event={"ID":"23af7f39-b236-441f-af8e-4c8a949f21cb","Type":"ContainerDied","Data":"57c99ba79d8f80365e0005b9ef3d955d5218389f926b55789b500c9a7812acd9"} Nov 03 22:33:05.662699 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.662632 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb" event={"ID":"23af7f39-b236-441f-af8e-4c8a949f21cb","Type":"ContainerDied","Data":"3df98fda2303715ee5f414ea9a3c84873cb1b6f6f337c32dd73dd33737b7627e"} Nov 03 22:33:05.662699 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.662663 5084 scope.go:122] "RemoveContainer" containerID="57c99ba79d8f80365e0005b9ef3d955d5218389f926b55789b500c9a7812acd9" Nov 03 22:33:05.711390 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.711359 5084 scope.go:122] "RemoveContainer" containerID="57c99ba79d8f80365e0005b9ef3d955d5218389f926b55789b500c9a7812acd9" Nov 03 22:33:05.711732 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:33:05.711708 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c99ba79d8f80365e0005b9ef3d955d5218389f926b55789b500c9a7812acd9\": container with ID starting with 57c99ba79d8f80365e0005b9ef3d955d5218389f926b55789b500c9a7812acd9 not found: ID does not exist" containerID="57c99ba79d8f80365e0005b9ef3d955d5218389f926b55789b500c9a7812acd9" Nov 03 22:33:05.711812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.711750 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c99ba79d8f80365e0005b9ef3d955d5218389f926b55789b500c9a7812acd9"} err="failed to get container status \"57c99ba79d8f80365e0005b9ef3d955d5218389f926b55789b500c9a7812acd9\": rpc error: code = NotFound desc = could not find container \"57c99ba79d8f80365e0005b9ef3d955d5218389f926b55789b500c9a7812acd9\": container with ID starting with 57c99ba79d8f80365e0005b9ef3d955d5218389f926b55789b500c9a7812acd9 not found: ID does not exist" Nov 03 22:33:05.713377 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:05.713338 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" podUID="63b8796a-e0c0-491c-a294-0b9eb689639c" containerName="payload-pre-processing" containerID="cri-o://87b74282896b09eafe11ccc52a8d4ab6de5236b6d753c64186c8e2cd2682e90e" gracePeriod=25 Nov 03 22:33:06.118287 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.118195 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb"] Nov 03 22:33:06.140474 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.140427 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e3937145-679f5895f5-qf7fb"] Nov 03 22:33:06.150808 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:33:06.150416 5084 logging.go:55] [core] [Channel #1181 SubChannel #1182]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.111:50052", ServerName: "10.132.0.111:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.111:50052: connect: connection refused" Nov 03 22:33:06.171600 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.171244 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" Nov 03 22:33:06.175711 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:33:06.175480 5084 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b8796a_e0c0_491c_a294_0b9eb689639c.slice/crio-87b74282896b09eafe11ccc52a8d4ab6de5236b6d753c64186c8e2cd2682e90e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b8796a_e0c0_491c_a294_0b9eb689639c.slice/crio-conmon-87b74282896b09eafe11ccc52a8d4ab6de5236b6d753c64186c8e2cd2682e90e.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:33:06.260281 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.260230 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a5fce82-2002-48a4-ba14-e5e4bf6bfed5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5fce82-2002-48a4-ba14-e5e4bf6bfed5-plugins-config-volume\") pod \"2a5fce82-2002-48a4-ba14-e5e4bf6bfed5\" (UID: \"2a5fce82-2002-48a4-ba14-e5e4bf6bfed5\") " Nov 03 22:33:06.260465 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.260414 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a5fce82-2002-48a4-ba14-e5e4bf6bfed5-kube-api-access-nlqk9\" (UniqueName: \"kubernetes.io/projected/2a5fce82-2002-48a4-ba14-e5e4bf6bfed5-kube-api-access-nlqk9\") pod \"2a5fce82-2002-48a4-ba14-e5e4bf6bfed5\" (UID: \"2a5fce82-2002-48a4-ba14-e5e4bf6bfed5\") " Nov 03 22:33:06.261390 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.261358 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5fce82-2002-48a4-ba14-e5e4bf6bfed5-plugins-config-volume" pod "2a5fce82-2002-48a4-ba14-e5e4bf6bfed5" (UID: "2a5fce82-2002-48a4-ba14-e5e4bf6bfed5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:06.268921 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.268880 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5fce82-2002-48a4-ba14-e5e4bf6bfed5-kube-api-access-nlqk9" pod "2a5fce82-2002-48a4-ba14-e5e4bf6bfed5" (UID: "2a5fce82-2002-48a4-ba14-e5e4bf6bfed5"). InnerVolumeSpecName "kube-api-access-nlqk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:06.340562 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.340529 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" Nov 03 22:33:06.362204 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.362165 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b8796a-e0c0-491c-a294-0b9eb689639c-kube-api-access-4g5pn\" (UniqueName: \"kubernetes.io/projected/63b8796a-e0c0-491c-a294-0b9eb689639c-kube-api-access-4g5pn\") pod \"63b8796a-e0c0-491c-a294-0b9eb689639c\" (UID: \"63b8796a-e0c0-491c-a294-0b9eb689639c\") " Nov 03 22:33:06.362770 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.362739 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63b8796a-e0c0-491c-a294-0b9eb689639c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b8796a-e0c0-491c-a294-0b9eb689639c-plugins-config-volume\") pod \"63b8796a-e0c0-491c-a294-0b9eb689639c\" (UID: \"63b8796a-e0c0-491c-a294-0b9eb689639c\") " Nov 03 22:33:06.363234 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.363205 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b8796a-e0c0-491c-a294-0b9eb689639c-plugins-config-volume" pod "63b8796a-e0c0-491c-a294-0b9eb689639c" (UID: "63b8796a-e0c0-491c-a294-0b9eb689639c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:06.363494 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.363472 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlqk9\" (UniqueName: \"kubernetes.io/projected/2a5fce82-2002-48a4-ba14-e5e4bf6bfed5-kube-api-access-nlqk9\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:06.363582 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.363500 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63b8796a-e0c0-491c-a294-0b9eb689639c-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:06.363582 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.363515 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5fce82-2002-48a4-ba14-e5e4bf6bfed5-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:06.364598 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.364571 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b8796a-e0c0-491c-a294-0b9eb689639c-kube-api-access-4g5pn" pod "63b8796a-e0c0-491c-a294-0b9eb689639c" (UID: "63b8796a-e0c0-491c-a294-0b9eb689639c"). InnerVolumeSpecName "kube-api-access-4g5pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:06.465378 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.465312 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g5pn\" (UniqueName: \"kubernetes.io/projected/63b8796a-e0c0-491c-a294-0b9eb689639c-kube-api-access-4g5pn\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:06.631741 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.631705 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23af7f39-b236-441f-af8e-4c8a949f21cb" path="/var/lib/kubelet/pods/23af7f39-b236-441f-af8e-4c8a949f21cb/volumes" Nov 03 22:33:06.632253 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.632237 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="705730b8-4c2d-42b2-b9df-005bf8005a15" path="/var/lib/kubelet/pods/705730b8-4c2d-42b2-b9df-005bf8005a15/volumes" Nov 03 22:33:06.676178 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.674959 5084 generic.go:356] "Generic (PLEG): container finished" podID="2a5fce82-2002-48a4-ba14-e5e4bf6bfed5" containerID="e61ffeac612291c8923b5bc0f444ff879f9ec99980ef7786565523779afb3427" exitCode=0 Nov 03 22:33:06.676178 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.675007 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" Nov 03 22:33:06.676178 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.675015 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" event={"ID":"2a5fce82-2002-48a4-ba14-e5e4bf6bfed5","Type":"ContainerDied","Data":"e61ffeac612291c8923b5bc0f444ff879f9ec99980ef7786565523779afb3427"} Nov 03 22:33:06.676178 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.675119 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" event={"ID":"2a5fce82-2002-48a4-ba14-e5e4bf6bfed5","Type":"ContainerDied","Data":"2db1e866c88ffd89f97c6cb1052fd83c2ffd1347fc2d75eb78d3e526bbc71829"} Nov 03 22:33:06.676178 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.675170 5084 scope.go:122] "RemoveContainer" containerID="e61ffeac612291c8923b5bc0f444ff879f9ec99980ef7786565523779afb3427" Nov 03 22:33:06.680469 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.680435 5084 generic.go:356] "Generic (PLEG): container finished" podID="63b8796a-e0c0-491c-a294-0b9eb689639c" containerID="87b74282896b09eafe11ccc52a8d4ab6de5236b6d753c64186c8e2cd2682e90e" exitCode=0 Nov 03 22:33:06.680716 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.680492 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" Nov 03 22:33:06.680716 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.680534 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" event={"ID":"63b8796a-e0c0-491c-a294-0b9eb689639c","Type":"ContainerDied","Data":"87b74282896b09eafe11ccc52a8d4ab6de5236b6d753c64186c8e2cd2682e90e"} Nov 03 22:33:06.680716 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.680609 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" event={"ID":"63b8796a-e0c0-491c-a294-0b9eb689639c","Type":"ContainerDied","Data":"6eb2ae439539c15faf8b8b18324de5e490401805bbf3fe3d534d105bfdba2dec"} Nov 03 22:33:06.696031 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.695991 5084 scope.go:122] "RemoveContainer" containerID="e61ffeac612291c8923b5bc0f444ff879f9ec99980ef7786565523779afb3427" Nov 03 22:33:06.696765 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:33:06.696726 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e61ffeac612291c8923b5bc0f444ff879f9ec99980ef7786565523779afb3427\": container with ID starting with e61ffeac612291c8923b5bc0f444ff879f9ec99980ef7786565523779afb3427 not found: ID does not exist" containerID="e61ffeac612291c8923b5bc0f444ff879f9ec99980ef7786565523779afb3427" Nov 03 22:33:06.696880 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.696775 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e61ffeac612291c8923b5bc0f444ff879f9ec99980ef7786565523779afb3427"} err="failed to get container status \"e61ffeac612291c8923b5bc0f444ff879f9ec99980ef7786565523779afb3427\": rpc error: code = NotFound desc = could not find container \"e61ffeac612291c8923b5bc0f444ff879f9ec99980ef7786565523779afb3427\": container with ID starting with e61ffeac612291c8923b5bc0f444ff879f9ec99980ef7786565523779afb3427 not found: ID does not exist" Nov 03 22:33:06.696880 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.696804 5084 scope.go:122] "RemoveContainer" containerID="87b74282896b09eafe11ccc52a8d4ab6de5236b6d753c64186c8e2cd2682e90e" Nov 03 22:33:06.708603 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.708577 5084 scope.go:122] "RemoveContainer" containerID="87b74282896b09eafe11ccc52a8d4ab6de5236b6d753c64186c8e2cd2682e90e" Nov 03 22:33:06.708957 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:33:06.708934 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b74282896b09eafe11ccc52a8d4ab6de5236b6d753c64186c8e2cd2682e90e\": container with ID starting with 87b74282896b09eafe11ccc52a8d4ab6de5236b6d753c64186c8e2cd2682e90e not found: ID does not exist" containerID="87b74282896b09eafe11ccc52a8d4ab6de5236b6d753c64186c8e2cd2682e90e" Nov 03 22:33:06.709047 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.708964 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b74282896b09eafe11ccc52a8d4ab6de5236b6d753c64186c8e2cd2682e90e"} err="failed to get container status \"87b74282896b09eafe11ccc52a8d4ab6de5236b6d753c64186c8e2cd2682e90e\": rpc error: code = NotFound desc = could not find container \"87b74282896b09eafe11ccc52a8d4ab6de5236b6d753c64186c8e2cd2682e90e\": container with ID starting with 87b74282896b09eafe11ccc52a8d4ab6de5236b6d753c64186c8e2cd2682e90e not found: ID does not exist" Nov 03 22:33:06.846720 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.846675 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj"] Nov 03 22:33:06.851347 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.851314 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj"] Nov 03 22:33:06.975268 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:06.975161 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2"] Nov 03 22:33:07.051790 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:07.051752 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2"] Nov 03 22:33:07.131540 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:07.131486 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-migrate-59a07a45-76d7565f4c-x4gkj" podUID="63b8796a-e0c0-491c-a294-0b9eb689639c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.111:50052\" within 1s: context deadline exceeded" Nov 03 22:33:07.132574 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:07.132538 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-migrate-59a07a45-797fcffcb4-2tlb2" podUID="2a5fce82-2002-48a4-ba14-e5e4bf6bfed5" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.110:50052\" within 1s: context deadline exceeded" Nov 03 22:33:07.132692 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:33:07.132668 5084 logging.go:55] [core] [Channel #1183 SubChannel #1184]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.110:50052", ServerName: "10.132.0.110:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.110:50052: operation was canceled" Nov 03 22:33:07.212483 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:07.212423 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc"] Nov 03 22:33:07.212750 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:07.212722 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" podUID="15696e53-b286-40ae-b2f2-fa6234b6426b" containerName="istio-proxy" containerID="cri-o://54c5eef7445896f46907fd3a58d759dec0088d8390afb8914eedc291ebd58639" gracePeriod=30 Nov 03 22:33:07.497087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:07.493595 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj" podUID="62161443-27f9-4dab-b2e1-b4616adf2374" containerName="payload-processing" containerID="cri-o://bb4dfa2197473e7c5e08a4f203428403a75fb91bf7fb89c8a73124910a1458ba" gracePeriod=25 Nov 03 22:33:07.560400 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:07.560155 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff" podUID="a37bdbda-c89d-43bf-84a7-6940116bc31f" containerName="payload-pre-processing" containerID="cri-o://8540340c764eb794741e842de71c3d2facc153943d06327bf234aa01babcd46b" gracePeriod=25 Nov 03 22:33:07.700899 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:07.700861 5084 generic.go:356] "Generic (PLEG): container finished" podID="62161443-27f9-4dab-b2e1-b4616adf2374" containerID="bb4dfa2197473e7c5e08a4f203428403a75fb91bf7fb89c8a73124910a1458ba" exitCode=0 Nov 03 22:33:07.701317 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:07.700979 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj" event={"ID":"62161443-27f9-4dab-b2e1-b4616adf2374","Type":"ContainerDied","Data":"bb4dfa2197473e7c5e08a4f203428403a75fb91bf7fb89c8a73124910a1458ba"} Nov 03 22:33:07.704737 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:07.704692 5084 generic.go:356] "Generic (PLEG): container finished" podID="a37bdbda-c89d-43bf-84a7-6940116bc31f" containerID="8540340c764eb794741e842de71c3d2facc153943d06327bf234aa01babcd46b" exitCode=0 Nov 03 22:33:07.704892 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:07.704755 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff" event={"ID":"a37bdbda-c89d-43bf-84a7-6940116bc31f","Type":"ContainerDied","Data":"8540340c764eb794741e842de71c3d2facc153943d06327bf234aa01babcd46b"} Nov 03 22:33:07.891105 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:07.886513 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj" Nov 03 22:33:07.900049 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:07.900015 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62161443-27f9-4dab-b2e1-b4616adf2374-kube-api-access-n5fqn\" (UniqueName: \"kubernetes.io/projected/62161443-27f9-4dab-b2e1-b4616adf2374-kube-api-access-n5fqn\") pod \"62161443-27f9-4dab-b2e1-b4616adf2374\" (UID: \"62161443-27f9-4dab-b2e1-b4616adf2374\") " Nov 03 22:33:07.900252 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:07.900148 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62161443-27f9-4dab-b2e1-b4616adf2374-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62161443-27f9-4dab-b2e1-b4616adf2374-plugins-config-volume\") pod \"62161443-27f9-4dab-b2e1-b4616adf2374\" (UID: \"62161443-27f9-4dab-b2e1-b4616adf2374\") " Nov 03 22:33:07.900589 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:07.900557 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62161443-27f9-4dab-b2e1-b4616adf2374-plugins-config-volume" pod "62161443-27f9-4dab-b2e1-b4616adf2374" (UID: "62161443-27f9-4dab-b2e1-b4616adf2374"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:07.900976 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:07.900861 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62161443-27f9-4dab-b2e1-b4616adf2374-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:07.902667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:07.902621 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62161443-27f9-4dab-b2e1-b4616adf2374-kube-api-access-n5fqn" pod "62161443-27f9-4dab-b2e1-b4616adf2374" (UID: "62161443-27f9-4dab-b2e1-b4616adf2374"). InnerVolumeSpecName "kube-api-access-n5fqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:07.914652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:07.914619 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff" Nov 03 22:33:08.001953 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.001905 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a37bdbda-c89d-43bf-84a7-6940116bc31f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37bdbda-c89d-43bf-84a7-6940116bc31f-plugins-config-volume\") pod \"a37bdbda-c89d-43bf-84a7-6940116bc31f\" (UID: \"a37bdbda-c89d-43bf-84a7-6940116bc31f\") " Nov 03 22:33:08.002211 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.002190 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37bdbda-c89d-43bf-84a7-6940116bc31f-kube-api-access-s2chz\" (UniqueName: \"kubernetes.io/projected/a37bdbda-c89d-43bf-84a7-6940116bc31f-kube-api-access-s2chz\") pod \"a37bdbda-c89d-43bf-84a7-6940116bc31f\" (UID: \"a37bdbda-c89d-43bf-84a7-6940116bc31f\") " Nov 03 22:33:08.002860 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.002832 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5fqn\" (UniqueName: \"kubernetes.io/projected/62161443-27f9-4dab-b2e1-b4616adf2374-kube-api-access-n5fqn\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:08.003202 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.003126 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37bdbda-c89d-43bf-84a7-6940116bc31f-plugins-config-volume" pod "a37bdbda-c89d-43bf-84a7-6940116bc31f" (UID: "a37bdbda-c89d-43bf-84a7-6940116bc31f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:08.005631 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.005558 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37bdbda-c89d-43bf-84a7-6940116bc31f-kube-api-access-s2chz" pod "a37bdbda-c89d-43bf-84a7-6940116bc31f" (UID: "a37bdbda-c89d-43bf-84a7-6940116bc31f"). InnerVolumeSpecName "kube-api-access-s2chz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:08.104758 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.104718 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2chz\" (UniqueName: \"kubernetes.io/projected/a37bdbda-c89d-43bf-84a7-6940116bc31f-kube-api-access-s2chz\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:08.104758 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.104754 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37bdbda-c89d-43bf-84a7-6940116bc31f-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:08.214821 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.214754 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh" podUID="8c7cdcde-5c77-491a-b84e-5f285ec4d794" containerName="payload-processing" containerID="cri-o://16d5f1104c1afd90a30c8f9ca2aa32763eb2c2bf3bf64757142b3cdb615ce73b" gracePeriod=25 Nov 03 22:33:08.505871 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.505837 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh" Nov 03 22:33:08.610222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.610117 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb" podUID="8ea29e4c-d210-48c0-b495-45958a542245" containerName="payload-pre-processing" containerID="cri-o://72585f2eb300fc71097746ee60ff826a2769c733c1c7a832db70c1315de9c88e" gracePeriod=25 Nov 03 22:33:08.610987 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.610961 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c7cdcde-5c77-491a-b84e-5f285ec4d794-kube-api-access-4mksq\" (UniqueName: \"kubernetes.io/projected/8c7cdcde-5c77-491a-b84e-5f285ec4d794-kube-api-access-4mksq\") pod \"8c7cdcde-5c77-491a-b84e-5f285ec4d794\" (UID: \"8c7cdcde-5c77-491a-b84e-5f285ec4d794\") " Nov 03 22:33:08.611342 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.611318 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c7cdcde-5c77-491a-b84e-5f285ec4d794-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c7cdcde-5c77-491a-b84e-5f285ec4d794-plugins-config-volume\") pod \"8c7cdcde-5c77-491a-b84e-5f285ec4d794\" (UID: \"8c7cdcde-5c77-491a-b84e-5f285ec4d794\") " Nov 03 22:33:08.611688 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.611660 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c7cdcde-5c77-491a-b84e-5f285ec4d794-plugins-config-volume" pod "8c7cdcde-5c77-491a-b84e-5f285ec4d794" (UID: "8c7cdcde-5c77-491a-b84e-5f285ec4d794"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:08.611943 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.611925 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c7cdcde-5c77-491a-b84e-5f285ec4d794-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:08.613310 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.613284 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7cdcde-5c77-491a-b84e-5f285ec4d794-kube-api-access-4mksq" pod "8c7cdcde-5c77-491a-b84e-5f285ec4d794" (UID: "8c7cdcde-5c77-491a-b84e-5f285ec4d794"). InnerVolumeSpecName "kube-api-access-4mksq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:08.630970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.630927 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a5fce82-2002-48a4-ba14-e5e4bf6bfed5" path="/var/lib/kubelet/pods/2a5fce82-2002-48a4-ba14-e5e4bf6bfed5/volumes" Nov 03 22:33:08.631476 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.631459 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63b8796a-e0c0-491c-a294-0b9eb689639c" path="/var/lib/kubelet/pods/63b8796a-e0c0-491c-a294-0b9eb689639c/volumes" Nov 03 22:33:08.713418 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.713372 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mksq\" (UniqueName: \"kubernetes.io/projected/8c7cdcde-5c77-491a-b84e-5f285ec4d794-kube-api-access-4mksq\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:08.717490 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.717254 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj" event={"ID":"62161443-27f9-4dab-b2e1-b4616adf2374","Type":"ContainerDied","Data":"23c9622914a2994c14248309109ca7ae4486e6af26ebc48b94a8026a41062448"} Nov 03 22:33:08.717490 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.717339 5084 scope.go:122] "RemoveContainer" containerID="bb4dfa2197473e7c5e08a4f203428403a75fb91bf7fb89c8a73124910a1458ba" Nov 03 22:33:08.717490 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.717265 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj" Nov 03 22:33:08.720311 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.719929 5084 generic.go:356] "Generic (PLEG): container finished" podID="8c7cdcde-5c77-491a-b84e-5f285ec4d794" containerID="16d5f1104c1afd90a30c8f9ca2aa32763eb2c2bf3bf64757142b3cdb615ce73b" exitCode=0 Nov 03 22:33:08.720311 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.720145 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh" event={"ID":"8c7cdcde-5c77-491a-b84e-5f285ec4d794","Type":"ContainerDied","Data":"16d5f1104c1afd90a30c8f9ca2aa32763eb2c2bf3bf64757142b3cdb615ce73b"} Nov 03 22:33:08.720311 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.720199 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh" event={"ID":"8c7cdcde-5c77-491a-b84e-5f285ec4d794","Type":"ContainerDied","Data":"dfb53689e4c54e1ebd3458532a427213844a9b5d87cf78b52d41fcb217c9d412"} Nov 03 22:33:08.720311 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.720153 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh" Nov 03 22:33:08.723442 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.723406 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff" event={"ID":"a37bdbda-c89d-43bf-84a7-6940116bc31f","Type":"ContainerDied","Data":"b477b566fb0188c587362df598693e3c0d268b9a61397d5514129321be6eea3e"} Nov 03 22:33:08.723641 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.723463 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff" Nov 03 22:33:08.731541 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.731517 5084 scope.go:122] "RemoveContainer" containerID="16d5f1104c1afd90a30c8f9ca2aa32763eb2c2bf3bf64757142b3cdb615ce73b" Nov 03 22:33:08.745812 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.745780 5084 scope.go:122] "RemoveContainer" containerID="16d5f1104c1afd90a30c8f9ca2aa32763eb2c2bf3bf64757142b3cdb615ce73b" Nov 03 22:33:08.746220 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:33:08.746198 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d5f1104c1afd90a30c8f9ca2aa32763eb2c2bf3bf64757142b3cdb615ce73b\": container with ID starting with 16d5f1104c1afd90a30c8f9ca2aa32763eb2c2bf3bf64757142b3cdb615ce73b not found: ID does not exist" containerID="16d5f1104c1afd90a30c8f9ca2aa32763eb2c2bf3bf64757142b3cdb615ce73b" Nov 03 22:33:08.746313 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.746233 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d5f1104c1afd90a30c8f9ca2aa32763eb2c2bf3bf64757142b3cdb615ce73b"} err="failed to get container status \"16d5f1104c1afd90a30c8f9ca2aa32763eb2c2bf3bf64757142b3cdb615ce73b\": rpc error: code = NotFound desc = could not find container \"16d5f1104c1afd90a30c8f9ca2aa32763eb2c2bf3bf64757142b3cdb615ce73b\": container with ID starting with 16d5f1104c1afd90a30c8f9ca2aa32763eb2c2bf3bf64757142b3cdb615ce73b not found: ID does not exist" Nov 03 22:33:08.746313 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.746259 5084 scope.go:122] "RemoveContainer" containerID="8540340c764eb794741e842de71c3d2facc153943d06327bf234aa01babcd46b" Nov 03 22:33:08.912036 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.911972 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff"] Nov 03 22:33:08.916808 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:08.916773 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-cee4c09d-7d9797b6b7-6h2ff"] Nov 03 22:33:09.026634 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.026561 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj"] Nov 03 22:33:09.036307 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.036049 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cee4c09d-5bc6fb4588-cmwmj"] Nov 03 22:33:09.074345 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.074308 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb" Nov 03 22:33:09.187232 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.187130 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh"] Nov 03 22:33:09.229830 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.229792 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e6e7831d-c767dbcd4-f5hgh"] Nov 03 22:33:09.253493 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.252806 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea29e4c-d210-48c0-b495-45958a542245-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea29e4c-d210-48c0-b495-45958a542245-plugins-config-volume\") pod \"8ea29e4c-d210-48c0-b495-45958a542245\" (UID: \"8ea29e4c-d210-48c0-b495-45958a542245\") " Nov 03 22:33:09.254077 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.254038 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea29e4c-d210-48c0-b495-45958a542245-kube-api-access-ll74q\" (UniqueName: \"kubernetes.io/projected/8ea29e4c-d210-48c0-b495-45958a542245-kube-api-access-ll74q\") pod \"8ea29e4c-d210-48c0-b495-45958a542245\" (UID: \"8ea29e4c-d210-48c0-b495-45958a542245\") " Nov 03 22:33:09.255679 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.253360 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea29e4c-d210-48c0-b495-45958a542245-plugins-config-volume" pod "8ea29e4c-d210-48c0-b495-45958a542245" (UID: "8ea29e4c-d210-48c0-b495-45958a542245"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:09.273098 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.268316 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea29e4c-d210-48c0-b495-45958a542245-kube-api-access-ll74q" pod "8ea29e4c-d210-48c0-b495-45958a542245" (UID: "8ea29e4c-d210-48c0-b495-45958a542245"). InnerVolumeSpecName "kube-api-access-ll74q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:09.363304 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.363264 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll74q\" (UniqueName: \"kubernetes.io/projected/8ea29e4c-d210-48c0-b495-45958a542245-kube-api-access-ll74q\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:09.374889 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.374853 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea29e4c-d210-48c0-b495-45958a542245-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:09.734538 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.734498 5084 generic.go:356] "Generic (PLEG): container finished" podID="8ea29e4c-d210-48c0-b495-45958a542245" containerID="72585f2eb300fc71097746ee60ff826a2769c733c1c7a832db70c1315de9c88e" exitCode=0 Nov 03 22:33:09.734988 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.734548 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb" Nov 03 22:33:09.734988 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.734557 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb" event={"ID":"8ea29e4c-d210-48c0-b495-45958a542245","Type":"ContainerDied","Data":"72585f2eb300fc71097746ee60ff826a2769c733c1c7a832db70c1315de9c88e"} Nov 03 22:33:09.734988 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.734619 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb" event={"ID":"8ea29e4c-d210-48c0-b495-45958a542245","Type":"ContainerDied","Data":"ba3ee7b362fdf13349d1844d6f78e146e039b14fa5d32bb6bd2d2917c25b7879"} Nov 03 22:33:09.734988 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.734647 5084 scope.go:122] "RemoveContainer" containerID="72585f2eb300fc71097746ee60ff826a2769c733c1c7a832db70c1315de9c88e" Nov 03 22:33:09.747280 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.747249 5084 scope.go:122] "RemoveContainer" containerID="72585f2eb300fc71097746ee60ff826a2769c733c1c7a832db70c1315de9c88e" Nov 03 22:33:09.747665 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:33:09.747635 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72585f2eb300fc71097746ee60ff826a2769c733c1c7a832db70c1315de9c88e\": container with ID starting with 72585f2eb300fc71097746ee60ff826a2769c733c1c7a832db70c1315de9c88e not found: ID does not exist" containerID="72585f2eb300fc71097746ee60ff826a2769c733c1c7a832db70c1315de9c88e" Nov 03 22:33:09.747761 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.747672 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72585f2eb300fc71097746ee60ff826a2769c733c1c7a832db70c1315de9c88e"} err="failed to get container status \"72585f2eb300fc71097746ee60ff826a2769c733c1c7a832db70c1315de9c88e\": rpc error: code = NotFound desc = could not find container \"72585f2eb300fc71097746ee60ff826a2769c733c1c7a832db70c1315de9c88e\": container with ID starting with 72585f2eb300fc71097746ee60ff826a2769c733c1c7a832db70c1315de9c88e not found: ID does not exist" Nov 03 22:33:09.907246 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.907136 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb"] Nov 03 22:33:09.955346 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:09.955303 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e6e7831d-5fc78cd987-bcrxb"] Nov 03 22:33:10.633541 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:10.633499 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62161443-27f9-4dab-b2e1-b4616adf2374" path="/var/lib/kubelet/pods/62161443-27f9-4dab-b2e1-b4616adf2374/volumes" Nov 03 22:33:10.634344 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:10.634319 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c7cdcde-5c77-491a-b84e-5f285ec4d794" path="/var/lib/kubelet/pods/8c7cdcde-5c77-491a-b84e-5f285ec4d794/volumes" Nov 03 22:33:10.634981 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:10.634959 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea29e4c-d210-48c0-b495-45958a542245" path="/var/lib/kubelet/pods/8ea29e4c-d210-48c0-b495-45958a542245/volumes" Nov 03 22:33:10.635642 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:10.635613 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a37bdbda-c89d-43bf-84a7-6940116bc31f" path="/var/lib/kubelet/pods/a37bdbda-c89d-43bf-84a7-6940116bc31f/volumes" Nov 03 22:33:11.454888 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:11.454846 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr"] Nov 03 22:33:11.461090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:11.456549 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" podUID="7a2a8e8f-3773-45fa-87dd-23445af497d7" containerName="maas-api" containerID="cri-o://d47356d6cad2921b6f1756d1177587a7d31a7b55fe69e4053119ebe64090be49" gracePeriod=30 Nov 03 22:33:11.930467 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:11.921413 5084 generic.go:356] "Generic (PLEG): container finished" podID="7a2a8e8f-3773-45fa-87dd-23445af497d7" containerID="d47356d6cad2921b6f1756d1177587a7d31a7b55fe69e4053119ebe64090be49" exitCode=0 Nov 03 22:33:11.930467 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:11.921488 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" event={"ID":"7a2a8e8f-3773-45fa-87dd-23445af497d7","Type":"ContainerDied","Data":"d47356d6cad2921b6f1756d1177587a7d31a7b55fe69e4053119ebe64090be49"} Nov 03 22:33:11.982104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:11.977664 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:33:12.085712 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.077522 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2a8e8f-3773-45fa-87dd-23445af497d7-kube-api-access-7h4hj\" (UniqueName: \"kubernetes.io/projected/7a2a8e8f-3773-45fa-87dd-23445af497d7-kube-api-access-7h4hj\") pod \"7a2a8e8f-3773-45fa-87dd-23445af497d7\" (UID: \"7a2a8e8f-3773-45fa-87dd-23445af497d7\") " Nov 03 22:33:12.085712 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.077904 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a2a8e8f-3773-45fa-87dd-23445af497d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a2a8e8f-3773-45fa-87dd-23445af497d7-maas-api-tls\") pod \"7a2a8e8f-3773-45fa-87dd-23445af497d7\" (UID: \"7a2a8e8f-3773-45fa-87dd-23445af497d7\") " Nov 03 22:33:12.085712 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.077983 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a2a8e8f-3773-45fa-87dd-23445af497d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a2a8e8f-3773-45fa-87dd-23445af497d7-metrics-certs\") pod \"7a2a8e8f-3773-45fa-87dd-23445af497d7\" (UID: \"7a2a8e8f-3773-45fa-87dd-23445af497d7\") " Nov 03 22:33:12.085712 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.078119 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2a8e8f-3773-45fa-87dd-23445af497d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2a8e8f-3773-45fa-87dd-23445af497d7-openshift-service-ca\") pod \"7a2a8e8f-3773-45fa-87dd-23445af497d7\" (UID: \"7a2a8e8f-3773-45fa-87dd-23445af497d7\") " Nov 03 22:33:12.094192 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.088981 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2a8e8f-3773-45fa-87dd-23445af497d7-openshift-service-ca" pod "7a2a8e8f-3773-45fa-87dd-23445af497d7" (UID: "7a2a8e8f-3773-45fa-87dd-23445af497d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:12.094192 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.089509 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2a8e8f-3773-45fa-87dd-23445af497d7-metrics-certs" pod "7a2a8e8f-3773-45fa-87dd-23445af497d7" (UID: "7a2a8e8f-3773-45fa-87dd-23445af497d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:33:12.127105 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.125908 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2a8e8f-3773-45fa-87dd-23445af497d7-maas-api-tls" pod "7a2a8e8f-3773-45fa-87dd-23445af497d7" (UID: "7a2a8e8f-3773-45fa-87dd-23445af497d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:33:12.150141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.148292 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2a8e8f-3773-45fa-87dd-23445af497d7-kube-api-access-7h4hj" pod "7a2a8e8f-3773-45fa-87dd-23445af497d7" (UID: "7a2a8e8f-3773-45fa-87dd-23445af497d7"). InnerVolumeSpecName "kube-api-access-7h4hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:12.177098 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:33:12.176223 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:33:12.180906 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.180017 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a2a8e8f-3773-45fa-87dd-23445af497d7-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:12.180906 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.180054 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a2a8e8f-3773-45fa-87dd-23445af497d7-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:12.180906 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.180088 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a2a8e8f-3773-45fa-87dd-23445af497d7-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:12.180906 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.180097 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h4hj\" (UniqueName: \"kubernetes.io/projected/7a2a8e8f-3773-45fa-87dd-23445af497d7-kube-api-access-7h4hj\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:12.928242 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.928212 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:33:12.935597 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.935555 5084 generic.go:356] "Generic (PLEG): container finished" podID="15696e53-b286-40ae-b2f2-fa6234b6426b" containerID="54c5eef7445896f46907fd3a58d759dec0088d8390afb8914eedc291ebd58639" exitCode=0 Nov 03 22:33:12.935816 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.935616 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" Nov 03 22:33:12.935816 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.935663 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" event={"ID":"15696e53-b286-40ae-b2f2-fa6234b6426b","Type":"ContainerDied","Data":"54c5eef7445896f46907fd3a58d759dec0088d8390afb8914eedc291ebd58639"} Nov 03 22:33:12.935816 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.935731 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc" event={"ID":"15696e53-b286-40ae-b2f2-fa6234b6426b","Type":"ContainerDied","Data":"16d5215754bd6c357b7b2a136592552a849fdb749b34be19cbf2e6474a781e7d"} Nov 03 22:33:12.935816 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.935767 5084 scope.go:122] "RemoveContainer" containerID="54c5eef7445896f46907fd3a58d759dec0088d8390afb8914eedc291ebd58639" Nov 03 22:33:12.938557 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.938497 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" event={"ID":"7a2a8e8f-3773-45fa-87dd-23445af497d7","Type":"ContainerDied","Data":"3f2044678409d0da39327bfb03d88a6d8fe5b0ebafb6468440afdf9ba3e65138"} Nov 03 22:33:12.938721 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.938629 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr" Nov 03 22:33:12.949890 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.949863 5084 scope.go:122] "RemoveContainer" containerID="e8f718ea0253d6a780a8f56d489a04e9a98b07a7dd45c8306057ab353a02dfde" Nov 03 22:33:12.963871 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.963841 5084 scope.go:122] "RemoveContainer" containerID="54c5eef7445896f46907fd3a58d759dec0088d8390afb8914eedc291ebd58639" Nov 03 22:33:12.964368 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:33:12.964336 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54c5eef7445896f46907fd3a58d759dec0088d8390afb8914eedc291ebd58639\": container with ID starting with 54c5eef7445896f46907fd3a58d759dec0088d8390afb8914eedc291ebd58639 not found: ID does not exist" containerID="54c5eef7445896f46907fd3a58d759dec0088d8390afb8914eedc291ebd58639" Nov 03 22:33:12.964484 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.964377 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c5eef7445896f46907fd3a58d759dec0088d8390afb8914eedc291ebd58639"} err="failed to get container status \"54c5eef7445896f46907fd3a58d759dec0088d8390afb8914eedc291ebd58639\": rpc error: code = NotFound desc = could not find container \"54c5eef7445896f46907fd3a58d759dec0088d8390afb8914eedc291ebd58639\": container with ID starting with 54c5eef7445896f46907fd3a58d759dec0088d8390afb8914eedc291ebd58639 not found: ID does not exist" Nov 03 22:33:12.964484 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.964404 5084 scope.go:122] "RemoveContainer" containerID="e8f718ea0253d6a780a8f56d489a04e9a98b07a7dd45c8306057ab353a02dfde" Nov 03 22:33:12.964756 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:33:12.964738 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f718ea0253d6a780a8f56d489a04e9a98b07a7dd45c8306057ab353a02dfde\": container with ID starting with e8f718ea0253d6a780a8f56d489a04e9a98b07a7dd45c8306057ab353a02dfde not found: ID does not exist" containerID="e8f718ea0253d6a780a8f56d489a04e9a98b07a7dd45c8306057ab353a02dfde" Nov 03 22:33:12.964833 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.964759 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f718ea0253d6a780a8f56d489a04e9a98b07a7dd45c8306057ab353a02dfde"} err="failed to get container status \"e8f718ea0253d6a780a8f56d489a04e9a98b07a7dd45c8306057ab353a02dfde\": rpc error: code = NotFound desc = could not find container \"e8f718ea0253d6a780a8f56d489a04e9a98b07a7dd45c8306057ab353a02dfde\": container with ID starting with e8f718ea0253d6a780a8f56d489a04e9a98b07a7dd45c8306057ab353a02dfde not found: ID does not exist" Nov 03 22:33:12.964833 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.964773 5084 scope.go:122] "RemoveContainer" containerID="d47356d6cad2921b6f1756d1177587a7d31a7b55fe69e4053119ebe64090be49" Nov 03 22:33:12.994970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.994877 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-workload-socket\") pod \"15696e53-b286-40ae-b2f2-fa6234b6426b\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " Nov 03 22:33:12.994970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.994932 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15696e53-b286-40ae-b2f2-fa6234b6426b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15696e53-b286-40ae-b2f2-fa6234b6426b-istiod-ca-cert\") pod \"15696e53-b286-40ae-b2f2-fa6234b6426b\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " Nov 03 22:33:12.994970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.994953 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-credential-socket\") pod \"15696e53-b286-40ae-b2f2-fa6234b6426b\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " Nov 03 22:33:12.995279 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.995005 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-envoy\") pod \"15696e53-b286-40ae-b2f2-fa6234b6426b\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " Nov 03 22:33:12.995279 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.995025 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-data\") pod \"15696e53-b286-40ae-b2f2-fa6234b6426b\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " Nov 03 22:33:12.995279 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.995056 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-podinfo\") pod \"15696e53-b286-40ae-b2f2-fa6234b6426b\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " Nov 03 22:33:12.995279 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.995145 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-token\" (UniqueName: \"kubernetes.io/projected/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-token\") pod \"15696e53-b286-40ae-b2f2-fa6234b6426b\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " Nov 03 22:33:12.995279 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.995209 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15696e53-b286-40ae-b2f2-fa6234b6426b-kube-api-access-ttlhl\" (UniqueName: \"kubernetes.io/projected/15696e53-b286-40ae-b2f2-fa6234b6426b-kube-api-access-ttlhl\") pod \"15696e53-b286-40ae-b2f2-fa6234b6426b\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " Nov 03 22:33:12.995279 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.995249 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-workload-certs\") pod \"15696e53-b286-40ae-b2f2-fa6234b6426b\" (UID: \"15696e53-b286-40ae-b2f2-fa6234b6426b\") " Nov 03 22:33:12.995596 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.995397 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-data" pod "15696e53-b286-40ae-b2f2-fa6234b6426b" (UID: "15696e53-b286-40ae-b2f2-fa6234b6426b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:12.995596 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.995415 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-credential-socket" pod "15696e53-b286-40ae-b2f2-fa6234b6426b" (UID: "15696e53-b286-40ae-b2f2-fa6234b6426b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:12.995688 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.995586 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-workload-socket" pod "15696e53-b286-40ae-b2f2-fa6234b6426b" (UID: "15696e53-b286-40ae-b2f2-fa6234b6426b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:12.995781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.995737 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-workload-certs" pod "15696e53-b286-40ae-b2f2-fa6234b6426b" (UID: "15696e53-b286-40ae-b2f2-fa6234b6426b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:12.996079 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.995497 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15696e53-b286-40ae-b2f2-fa6234b6426b-istiod-ca-cert" pod "15696e53-b286-40ae-b2f2-fa6234b6426b" (UID: "15696e53-b286-40ae-b2f2-fa6234b6426b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:12.996366 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.996343 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:12.996450 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.996373 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15696e53-b286-40ae-b2f2-fa6234b6426b-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:12.996450 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.996390 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:12.996450 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.996405 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:12.996450 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.996420 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:12.997552 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.997524 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-podinfo" pod "15696e53-b286-40ae-b2f2-fa6234b6426b" (UID: "15696e53-b286-40ae-b2f2-fa6234b6426b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:33:12.997679 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.997659 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-envoy" pod "15696e53-b286-40ae-b2f2-fa6234b6426b" (UID: "15696e53-b286-40ae-b2f2-fa6234b6426b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:12.997754 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.997671 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15696e53-b286-40ae-b2f2-fa6234b6426b-kube-api-access-ttlhl" pod "15696e53-b286-40ae-b2f2-fa6234b6426b" (UID: "15696e53-b286-40ae-b2f2-fa6234b6426b"). InnerVolumeSpecName "kube-api-access-ttlhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:12.997754 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:12.997716 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-token" pod "15696e53-b286-40ae-b2f2-fa6234b6426b" (UID: "15696e53-b286-40ae-b2f2-fa6234b6426b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:13.098039 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:13.098003 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:13.098039 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:13.098034 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:13.098039 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:13.098046 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15696e53-b286-40ae-b2f2-fa6234b6426b-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:13.098315 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:13.098055 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttlhl\" (UniqueName: \"kubernetes.io/projected/15696e53-b286-40ae-b2f2-fa6234b6426b-kube-api-access-ttlhl\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:13.293088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:13.290140 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr"] Nov 03 22:33:13.306000 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:13.305934 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-59a07a45-78dfc8778-85vxr"] Nov 03 22:33:13.419787 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:13.419748 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc"] Nov 03 22:33:13.424313 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:13.424272 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e3937145-openshift-default-6c97d4f5b9-nc9dc"] Nov 03 22:33:14.633190 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:14.633149 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15696e53-b286-40ae-b2f2-fa6234b6426b" path="/var/lib/kubelet/pods/15696e53-b286-40ae-b2f2-fa6234b6426b/volumes" Nov 03 22:33:14.633693 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:14.633676 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2a8e8f-3773-45fa-87dd-23445af497d7" path="/var/lib/kubelet/pods/7a2a8e8f-3773-45fa-87dd-23445af497d7/volumes" Nov 03 22:33:15.051099 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:15.051028 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r"] Nov 03 22:33:15.051397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:15.051360 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" podUID="3afecb8a-983a-4938-a155-c9561b7c1144" containerName="istio-proxy" containerID="cri-o://7609f13f6d28f1ccb379d04a7b0cdebc265bb27f3b7a6fab34a18c817e938dd8" gracePeriod=30 Nov 03 22:33:16.053922 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.053881 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs"] Nov 03 22:33:16.054571 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054552 5084 cpu_manager.go:414] "containerMap: removing container" podUID="a37bdbda-c89d-43bf-84a7-6940116bc31f" containerName="payload-pre-processing" Nov 03 22:33:16.054610 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054572 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a37bdbda-c89d-43bf-84a7-6940116bc31f" containerName="payload-pre-processing" Nov 03 22:33:16.054610 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054589 5084 cpu_manager.go:414] "containerMap: removing container" podUID="705730b8-4c2d-42b2-b9df-005bf8005a15" containerName="maas-api" Nov 03 22:33:16.054610 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054594 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705730b8-4c2d-42b2-b9df-005bf8005a15" containerName="maas-api" Nov 03 22:33:16.054610 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054603 5084 cpu_manager.go:414] "containerMap: removing container" podUID="7a2a8e8f-3773-45fa-87dd-23445af497d7" containerName="maas-api" Nov 03 22:33:16.054610 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054607 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2a8e8f-3773-45fa-87dd-23445af497d7" containerName="maas-api" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054623 5084 cpu_manager.go:414] "containerMap: removing container" podUID="62161443-27f9-4dab-b2e1-b4616adf2374" containerName="payload-processing" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054627 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62161443-27f9-4dab-b2e1-b4616adf2374" containerName="payload-processing" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054636 5084 cpu_manager.go:414] "containerMap: removing container" podUID="4becc34f-f258-4cae-b0fd-4cb837a5b2ed" containerName="maas-api" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054641 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4becc34f-f258-4cae-b0fd-4cb837a5b2ed" containerName="maas-api" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054650 5084 cpu_manager.go:414] "containerMap: removing container" podUID="23af7f39-b236-441f-af8e-4c8a949f21cb" containerName="maas-api" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054655 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23af7f39-b236-441f-af8e-4c8a949f21cb" containerName="maas-api" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054661 5084 cpu_manager.go:414] "containerMap: removing container" podUID="1ec48958-96f5-4e49-a495-df51c0c5c029" containerName="payload-processing" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054665 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec48958-96f5-4e49-a495-df51c0c5c029" containerName="payload-processing" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054673 5084 cpu_manager.go:414] "containerMap: removing container" podUID="2a5fce82-2002-48a4-ba14-e5e4bf6bfed5" containerName="payload-processing" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054677 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5fce82-2002-48a4-ba14-e5e4bf6bfed5" containerName="payload-processing" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054694 5084 cpu_manager.go:414] "containerMap: removing container" podUID="15696e53-b286-40ae-b2f2-fa6234b6426b" containerName="istio-proxy" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054699 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15696e53-b286-40ae-b2f2-fa6234b6426b" containerName="istio-proxy" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054705 5084 cpu_manager.go:414] "containerMap: removing container" podUID="15696e53-b286-40ae-b2f2-fa6234b6426b" containerName="istio-proxy" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054710 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15696e53-b286-40ae-b2f2-fa6234b6426b" containerName="istio-proxy" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054721 5084 cpu_manager.go:414] "containerMap: removing container" podUID="8c7cdcde-5c77-491a-b84e-5f285ec4d794" containerName="payload-processing" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054725 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c7cdcde-5c77-491a-b84e-5f285ec4d794" containerName="payload-processing" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054733 5084 cpu_manager.go:414] "containerMap: removing container" podUID="63b8796a-e0c0-491c-a294-0b9eb689639c" containerName="payload-pre-processing" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054737 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b8796a-e0c0-491c-a294-0b9eb689639c" containerName="payload-pre-processing" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054745 5084 cpu_manager.go:414] "containerMap: removing container" podUID="8ea29e4c-d210-48c0-b495-45958a542245" containerName="payload-pre-processing" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054750 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea29e4c-d210-48c0-b495-45958a542245" containerName="payload-pre-processing" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054759 5084 cpu_manager.go:414] "containerMap: removing container" podUID="2eb841e6-f440-4cf4-a36b-4604c2bc92c3" containerName="payload-pre-processing" Nov 03 22:33:16.054781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054763 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb841e6-f440-4cf4-a36b-4604c2bc92c3" containerName="payload-pre-processing" Nov 03 22:33:16.055462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054860 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="23af7f39-b236-441f-af8e-4c8a949f21cb" containerName="maas-api" Nov 03 22:33:16.055462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054868 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec48958-96f5-4e49-a495-df51c0c5c029" containerName="payload-processing" Nov 03 22:33:16.055462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054876 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5fce82-2002-48a4-ba14-e5e4bf6bfed5" containerName="payload-processing" Nov 03 22:33:16.055462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054886 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="15696e53-b286-40ae-b2f2-fa6234b6426b" containerName="istio-proxy" Nov 03 22:33:16.055462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054892 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="15696e53-b286-40ae-b2f2-fa6234b6426b" containerName="istio-proxy" Nov 03 22:33:16.055462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054900 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c7cdcde-5c77-491a-b84e-5f285ec4d794" containerName="payload-processing" Nov 03 22:33:16.055462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054906 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b8796a-e0c0-491c-a294-0b9eb689639c" containerName="payload-pre-processing" Nov 03 22:33:16.055462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054914 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea29e4c-d210-48c0-b495-45958a542245" containerName="payload-pre-processing" Nov 03 22:33:16.055462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054920 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb841e6-f440-4cf4-a36b-4604c2bc92c3" containerName="payload-pre-processing" Nov 03 22:33:16.055462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054926 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="a37bdbda-c89d-43bf-84a7-6940116bc31f" containerName="payload-pre-processing" Nov 03 22:33:16.055462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054932 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="705730b8-4c2d-42b2-b9df-005bf8005a15" containerName="maas-api" Nov 03 22:33:16.055462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054936 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2a8e8f-3773-45fa-87dd-23445af497d7" containerName="maas-api" Nov 03 22:33:16.055462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054942 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="62161443-27f9-4dab-b2e1-b4616adf2374" containerName="payload-processing" Nov 03 22:33:16.055462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.054948 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="4becc34f-f258-4cae-b0fd-4cb837a5b2ed" containerName="maas-api" Nov 03 22:33:16.060889 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.060857 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.131297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.115291 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-workload-certs\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.131297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.115352 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istiod-ca-cert\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.131297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.115395 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kslf\" (UniqueName: \"kubernetes.io/projected/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-kube-api-access-6kslf\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.131297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.115426 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-podinfo\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.131297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.115499 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-workload-socket\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.131297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.115615 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-token\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.131297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.115691 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-credential-socket\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.131297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.115859 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-envoy\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.131297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.115924 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-data\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.228938 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.228903 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c211881d-openshift-default-dockercfg-gltlf\"" Nov 03 22:33:16.288591 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.288546 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-data\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.288845 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.288704 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-workload-certs\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.288845 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.288743 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istiod-ca-cert\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.288845 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.288777 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kslf\" (UniqueName: \"kubernetes.io/projected/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-kube-api-access-6kslf\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.288845 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.288807 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-podinfo\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.299498 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.288881 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-workload-socket\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.299498 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.288988 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-token\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.299498 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.289042 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-credential-socket\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.299498 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.289237 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-envoy\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.361332 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.358954 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-credential-socket\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.361332 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.359419 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istiod-ca-cert\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.361843 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.361703 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-podinfo\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.361975 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.361883 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-workload-certs\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.371084 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.370952 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs"] Nov 03 22:33:16.374682 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.374328 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-workload-socket\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.374682 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.374563 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-envoy\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.407030 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.375707 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-data\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.407030 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.402263 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-token\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.434745 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.434665 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kslf\" (UniqueName: \"kubernetes.io/projected/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-kube-api-access-6kslf\") pod \"e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:16.558905 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:16.558593 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:17.152395 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:33:17.152351 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e734ea_1f54_4603_9fd9_3261ba6d4ec7.slice/crio-693d906d04ae621102ce255227b927909d3608fa99d812e917b7844e592ca96f WatchSource:0}: Error finding container 693d906d04ae621102ce255227b927909d3608fa99d812e917b7844e592ca96f: Status 404 returned error can't find the container with id 693d906d04ae621102ce255227b927909d3608fa99d812e917b7844e592ca96f Nov 03 22:33:17.152796 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:17.152580 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs"] Nov 03 22:33:17.154833 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:17.154791 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:33:17.154966 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:17.154876 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:33:17.154966 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:17.154921 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:33:17.598085 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:17.595112 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z"] Nov 03 22:33:17.635268 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:17.635217 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:17.648810 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:17.648721 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z"] Nov 03 22:33:17.801038 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:17.800507 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bc88eb47-openshift-default-dockercfg-h27hg\"" Nov 03 22:33:17.907655 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:17.907619 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:17.917092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:17.908136 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:17.917483 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:17.917451 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:17.917709 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:17.917688 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9zh\" (UniqueName: \"kubernetes.io/projected/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-kube-api-access-cq9zh\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:17.917859 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:17.917839 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:17.918032 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:17.918013 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:17.918165 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:17.918149 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:17.921734 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:17.918427 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:17.934962 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:17.934919 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.037622 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.037577 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.038526 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.038500 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.039178 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.039153 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9zh\" (UniqueName: \"kubernetes.io/projected/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-kube-api-access-cq9zh\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.039703 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.038361 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.039850 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.039673 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.040319 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.040300 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.040724 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.040708 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.041366 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.041345 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.041840 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.041821 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.043972 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.043950 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.125297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.041736 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.125297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.040216 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.125297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.043808 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.125297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.040656 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.125620 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.094435 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.125746 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.125707 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.131463 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.130411 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" event={"ID":"18e734ea-1f54-4603-9fd9-3261ba6d4ec7","Type":"ContainerStarted","Data":"a5d01c285a29b9b9289e19676165eafd86047db93daacdc38be306402a49aaba"} Nov 03 22:33:18.131463 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.130480 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" event={"ID":"18e734ea-1f54-4603-9fd9-3261ba6d4ec7","Type":"ContainerStarted","Data":"693d906d04ae621102ce255227b927909d3608fa99d812e917b7844e592ca96f"} Nov 03 22:33:18.132710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.132674 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9zh\" (UniqueName: \"kubernetes.io/projected/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-kube-api-access-cq9zh\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.133816 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.133788 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.207129 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.207044 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:18.253955 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.253897 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" podStartSLOduration=2.253882523 podStartE2EDuration="2.253882523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:18.251136976 +0000 UTC m=+1565.845169156" watchObservedRunningTime="2026-11-03 22:33:18.253882523 +0000 UTC m=+1565.847914494" Nov 03 22:33:18.563418 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.563373 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:18.569406 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.569375 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:18.679613 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:33:18.679569 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad18cacf_d8cc_4b0d_909e_7c60805abe2f.slice/crio-a40607bc4691913b340f491fa2f80df54104f5c63342c159a332eb75eae3dd44 WatchSource:0}: Error finding container a40607bc4691913b340f491fa2f80df54104f5c63342c159a332eb75eae3dd44: Status 404 returned error can't find the container with id a40607bc4691913b340f491fa2f80df54104f5c63342c159a332eb75eae3dd44 Nov 03 22:33:18.680341 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.680196 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z"] Nov 03 22:33:18.682550 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.682508 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:33:18.682672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.682599 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:33:18.682672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:18.682638 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:33:19.140253 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:19.140210 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" event={"ID":"ad18cacf-d8cc-4b0d-909e-7c60805abe2f","Type":"ContainerStarted","Data":"882ba25e5635fb50655d1c4f91fda96eae73ffe1b16a2731aaccc522a217aa93"} Nov 03 22:33:19.140452 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:19.140279 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" event={"ID":"ad18cacf-d8cc-4b0d-909e-7c60805abe2f","Type":"ContainerStarted","Data":"a40607bc4691913b340f491fa2f80df54104f5c63342c159a332eb75eae3dd44"} Nov 03 22:33:19.140545 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:19.140493 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:19.142171 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:19.142142 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:33:19.207695 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:19.207656 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:19.213468 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:19.213441 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:19.264552 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:19.264489 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" podStartSLOduration=2.264470325 podStartE2EDuration="2.264470325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:19.259960714 +0000 UTC m=+1566.853992688" watchObservedRunningTime="2026-11-03 22:33:19.264470325 +0000 UTC m=+1566.858502299" Nov 03 22:33:20.152056 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.152016 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:20.153022 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.152999 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:33:20.382896 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.382863 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:33:20.490762 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.490721 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-workload-socket\") pod \"3afecb8a-983a-4938-a155-c9561b7c1144\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " Nov 03 22:33:20.490996 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.490816 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3afecb8a-983a-4938-a155-c9561b7c1144-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afecb8a-983a-4938-a155-c9561b7c1144-istio-podinfo\") pod \"3afecb8a-983a-4938-a155-c9561b7c1144\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " Nov 03 22:33:20.490996 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.490934 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-istio-data\") pod \"3afecb8a-983a-4938-a155-c9561b7c1144\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " Nov 03 22:33:20.491143 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.491000 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3afecb8a-983a-4938-a155-c9561b7c1144-istio-token\" (UniqueName: \"kubernetes.io/projected/3afecb8a-983a-4938-a155-c9561b7c1144-istio-token\") pod \"3afecb8a-983a-4938-a155-c9561b7c1144\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " Nov 03 22:33:20.491143 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.491052 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-istio-envoy\") pod \"3afecb8a-983a-4938-a155-c9561b7c1144\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " Nov 03 22:33:20.491143 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.491112 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3afecb8a-983a-4938-a155-c9561b7c1144-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afecb8a-983a-4938-a155-c9561b7c1144-istiod-ca-cert\") pod \"3afecb8a-983a-4938-a155-c9561b7c1144\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " Nov 03 22:33:20.491305 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.491164 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-workload-certs\") pod \"3afecb8a-983a-4938-a155-c9561b7c1144\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " Nov 03 22:33:20.491305 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.491192 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-credential-socket\") pod \"3afecb8a-983a-4938-a155-c9561b7c1144\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " Nov 03 22:33:20.491305 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.491255 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3afecb8a-983a-4938-a155-c9561b7c1144-kube-api-access-6ppwm\" (UniqueName: \"kubernetes.io/projected/3afecb8a-983a-4938-a155-c9561b7c1144-kube-api-access-6ppwm\") pod \"3afecb8a-983a-4938-a155-c9561b7c1144\" (UID: \"3afecb8a-983a-4938-a155-c9561b7c1144\") " Nov 03 22:33:20.491509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.491155 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-workload-socket" pod "3afecb8a-983a-4938-a155-c9561b7c1144" (UID: "3afecb8a-983a-4938-a155-c9561b7c1144"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:20.491576 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.491508 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afecb8a-983a-4938-a155-c9561b7c1144-istiod-ca-cert" pod "3afecb8a-983a-4938-a155-c9561b7c1144" (UID: "3afecb8a-983a-4938-a155-c9561b7c1144"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:20.492051 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.491808 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-workload-certs" pod "3afecb8a-983a-4938-a155-c9561b7c1144" (UID: "3afecb8a-983a-4938-a155-c9561b7c1144"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:20.492051 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.492008 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:20.492051 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.492023 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afecb8a-983a-4938-a155-c9561b7c1144-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:20.492051 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.492033 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:20.492441 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.492417 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-credential-socket" pod "3afecb8a-983a-4938-a155-c9561b7c1144" (UID: "3afecb8a-983a-4938-a155-c9561b7c1144"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:20.492657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.492639 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-istio-data" pod "3afecb8a-983a-4938-a155-c9561b7c1144" (UID: "3afecb8a-983a-4938-a155-c9561b7c1144"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:20.493855 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.493818 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3afecb8a-983a-4938-a155-c9561b7c1144-istio-podinfo" pod "3afecb8a-983a-4938-a155-c9561b7c1144" (UID: "3afecb8a-983a-4938-a155-c9561b7c1144"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:33:20.493982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.493956 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-istio-envoy" pod "3afecb8a-983a-4938-a155-c9561b7c1144" (UID: "3afecb8a-983a-4938-a155-c9561b7c1144"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:33:20.494199 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.494179 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afecb8a-983a-4938-a155-c9561b7c1144-istio-token" pod "3afecb8a-983a-4938-a155-c9561b7c1144" (UID: "3afecb8a-983a-4938-a155-c9561b7c1144"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:20.494297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.494282 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afecb8a-983a-4938-a155-c9561b7c1144-kube-api-access-6ppwm" pod "3afecb8a-983a-4938-a155-c9561b7c1144" (UID: "3afecb8a-983a-4938-a155-c9561b7c1144"). InnerVolumeSpecName "kube-api-access-6ppwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:20.593956 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.593847 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:20.593956 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.593886 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ppwm\" (UniqueName: \"kubernetes.io/projected/3afecb8a-983a-4938-a155-c9561b7c1144-kube-api-access-6ppwm\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:20.593956 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.593898 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afecb8a-983a-4938-a155-c9561b7c1144-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:20.593956 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.593907 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:20.593956 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.593916 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afecb8a-983a-4938-a155-c9561b7c1144-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:20.593956 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:20.593924 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afecb8a-983a-4938-a155-c9561b7c1144-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:21.161918 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:21.161881 5084 generic.go:356] "Generic (PLEG): container finished" podID="3afecb8a-983a-4938-a155-c9561b7c1144" containerID="7609f13f6d28f1ccb379d04a7b0cdebc265bb27f3b7a6fab34a18c817e938dd8" exitCode=0 Nov 03 22:33:21.162119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:21.161960 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" Nov 03 22:33:21.162119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:21.161964 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" event={"ID":"3afecb8a-983a-4938-a155-c9561b7c1144","Type":"ContainerDied","Data":"7609f13f6d28f1ccb379d04a7b0cdebc265bb27f3b7a6fab34a18c817e938dd8"} Nov 03 22:33:21.162119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:21.162035 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r" event={"ID":"3afecb8a-983a-4938-a155-c9561b7c1144","Type":"ContainerDied","Data":"0fee6a945376c72f7394e05d51b7a6aa6c3ce3a60b845fcebef3ee5368c766f6"} Nov 03 22:33:21.162119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:21.162079 5084 scope.go:122] "RemoveContainer" containerID="7609f13f6d28f1ccb379d04a7b0cdebc265bb27f3b7a6fab34a18c817e938dd8" Nov 03 22:33:21.173876 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:21.173852 5084 scope.go:122] "RemoveContainer" containerID="f444bbf9c1bcce1f1e97471f788b3d795fdbe633222732dc0675131462755ca2" Nov 03 22:33:21.185801 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:21.185623 5084 scope.go:122] "RemoveContainer" containerID="7609f13f6d28f1ccb379d04a7b0cdebc265bb27f3b7a6fab34a18c817e938dd8" Nov 03 22:33:21.186112 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:33:21.186086 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7609f13f6d28f1ccb379d04a7b0cdebc265bb27f3b7a6fab34a18c817e938dd8\": container with ID starting with 7609f13f6d28f1ccb379d04a7b0cdebc265bb27f3b7a6fab34a18c817e938dd8 not found: ID does not exist" containerID="7609f13f6d28f1ccb379d04a7b0cdebc265bb27f3b7a6fab34a18c817e938dd8" Nov 03 22:33:21.186211 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:21.186120 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7609f13f6d28f1ccb379d04a7b0cdebc265bb27f3b7a6fab34a18c817e938dd8"} err="failed to get container status \"7609f13f6d28f1ccb379d04a7b0cdebc265bb27f3b7a6fab34a18c817e938dd8\": rpc error: code = NotFound desc = could not find container \"7609f13f6d28f1ccb379d04a7b0cdebc265bb27f3b7a6fab34a18c817e938dd8\": container with ID starting with 7609f13f6d28f1ccb379d04a7b0cdebc265bb27f3b7a6fab34a18c817e938dd8 not found: ID does not exist" Nov 03 22:33:21.186211 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:21.186138 5084 scope.go:122] "RemoveContainer" containerID="f444bbf9c1bcce1f1e97471f788b3d795fdbe633222732dc0675131462755ca2" Nov 03 22:33:21.186446 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:33:21.186417 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f444bbf9c1bcce1f1e97471f788b3d795fdbe633222732dc0675131462755ca2\": container with ID starting with f444bbf9c1bcce1f1e97471f788b3d795fdbe633222732dc0675131462755ca2 not found: ID does not exist" containerID="f444bbf9c1bcce1f1e97471f788b3d795fdbe633222732dc0675131462755ca2" Nov 03 22:33:21.186484 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:21.186454 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f444bbf9c1bcce1f1e97471f788b3d795fdbe633222732dc0675131462755ca2"} err="failed to get container status \"f444bbf9c1bcce1f1e97471f788b3d795fdbe633222732dc0675131462755ca2\": rpc error: code = NotFound desc = could not find container \"f444bbf9c1bcce1f1e97471f788b3d795fdbe633222732dc0675131462755ca2\": container with ID starting with f444bbf9c1bcce1f1e97471f788b3d795fdbe633222732dc0675131462755ca2 not found: ID does not exist" Nov 03 22:33:21.348513 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:21.348474 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r"] Nov 03 22:33:21.399997 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:21.399957 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e6e7831d-openshift-default-69f457ff9-tjq2r"] Nov 03 22:33:22.631240 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:22.631161 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3afecb8a-983a-4938-a155-c9561b7c1144" path="/var/lib/kubelet/pods/3afecb8a-983a-4938-a155-c9561b7c1144/volumes" Nov 03 22:33:22.895002 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:22.894880 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s"] Nov 03 22:33:27.784489 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:27.742127 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw"] Nov 03 22:33:27.784489 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:27.743935 5084 cpu_manager.go:414] "containerMap: removing container" podUID="3afecb8a-983a-4938-a155-c9561b7c1144" containerName="istio-proxy" Nov 03 22:33:27.784489 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:27.743965 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3afecb8a-983a-4938-a155-c9561b7c1144" containerName="istio-proxy" Nov 03 22:33:27.784489 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:27.744048 5084 cpu_manager.go:414] "containerMap: removing container" podUID="3afecb8a-983a-4938-a155-c9561b7c1144" containerName="istio-proxy" Nov 03 22:33:27.784489 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:27.744057 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3afecb8a-983a-4938-a155-c9561b7c1144" containerName="istio-proxy" Nov 03 22:33:27.784489 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:27.744892 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="3afecb8a-983a-4938-a155-c9561b7c1144" containerName="istio-proxy" Nov 03 22:33:27.886848 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:27.886811 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw"] Nov 03 22:33:27.887199 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:27.887177 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f"] Nov 03 22:33:27.890813 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:27.890767 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="3afecb8a-983a-4938-a155-c9561b7c1144" containerName="istio-proxy" Nov 03 22:33:27.901175 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:27.899241 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw" Nov 03 22:33:27.905122 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:27.905042 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f"] Nov 03 22:33:27.905508 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:27.905486 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f" Nov 03 22:33:27.920890 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:27.909904 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa113bb-18e1-4bd6-9c24-65e16319bf5a-plugins-config-volume\") pod \"payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw\" (UID: \"eaa113bb-18e1-4bd6-9c24-65e16319bf5a\") " pod="openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw" Nov 03 22:33:27.920890 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:27.909988 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfss7\" (UniqueName: \"kubernetes.io/projected/eaa113bb-18e1-4bd6-9c24-65e16319bf5a-kube-api-access-cfss7\") pod \"payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw\" (UID: \"eaa113bb-18e1-4bd6-9c24-65e16319bf5a\") " pod="openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw" Nov 03 22:33:27.920890 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:27.901020 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s" podUID="6462ea50-4efc-40bf-8775-39dd4179bd5e" containerName="payload-processing" containerID="cri-o://0c21a6c6bf2a5ef7577233753892d3ab46a6c68b83020e4e5e71d50e9a2ac70a" gracePeriod=25 Nov 03 22:33:28.031091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.008153 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c211881d-dockercfg-gtwx8\"" Nov 03 22:33:28.039594 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.031502 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c211881d\"" Nov 03 22:33:28.048972 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.047297 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5b4d\" (UniqueName: \"kubernetes.io/projected/5c54e202-d710-4a52-8c33-f773bdd018b5-kube-api-access-p5b4d\") pod \"payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f\" (UID: \"5c54e202-d710-4a52-8c33-f773bdd018b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f" Nov 03 22:33:28.051161 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.049339 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c54e202-d710-4a52-8c33-f773bdd018b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f\" (UID: \"5c54e202-d710-4a52-8c33-f773bdd018b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f" Nov 03 22:33:28.076113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.054494 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa113bb-18e1-4bd6-9c24-65e16319bf5a-plugins-config-volume\") pod \"payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw\" (UID: \"eaa113bb-18e1-4bd6-9c24-65e16319bf5a\") " pod="openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw" Nov 03 22:33:28.319394 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.062389 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa113bb-18e1-4bd6-9c24-65e16319bf5a-plugins-config-volume\") pod \"payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw\" (UID: \"eaa113bb-18e1-4bd6-9c24-65e16319bf5a\") " pod="openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw" Nov 03 22:33:28.319722 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.302543 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n"] Nov 03 22:33:28.319992 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.319955 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfss7\" (UniqueName: \"kubernetes.io/projected/eaa113bb-18e1-4bd6-9c24-65e16319bf5a-kube-api-access-cfss7\") pod \"payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw\" (UID: \"eaa113bb-18e1-4bd6-9c24-65e16319bf5a\") " pod="openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw" Nov 03 22:33:28.322644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.322582 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12a34bd6-bj45n"] Nov 03 22:33:28.378148 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.378047 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfss7\" (UniqueName: \"kubernetes.io/projected/eaa113bb-18e1-4bd6-9c24-65e16319bf5a-kube-api-access-cfss7\") pod \"payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw\" (UID: \"eaa113bb-18e1-4bd6-9c24-65e16319bf5a\") " pod="openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw" Nov 03 22:33:28.415451 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.415412 5084 generic.go:356] "Generic (PLEG): container finished" podID="6462ea50-4efc-40bf-8775-39dd4179bd5e" containerID="0c21a6c6bf2a5ef7577233753892d3ab46a6c68b83020e4e5e71d50e9a2ac70a" exitCode=0 Nov 03 22:33:28.415596 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.415483 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s" event={"ID":"6462ea50-4efc-40bf-8775-39dd4179bd5e","Type":"ContainerDied","Data":"0c21a6c6bf2a5ef7577233753892d3ab46a6c68b83020e4e5e71d50e9a2ac70a"} Nov 03 22:33:28.422407 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.422367 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5b4d\" (UniqueName: \"kubernetes.io/projected/5c54e202-d710-4a52-8c33-f773bdd018b5-kube-api-access-p5b4d\") pod \"payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f\" (UID: \"5c54e202-d710-4a52-8c33-f773bdd018b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f" Nov 03 22:33:28.422600 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.422426 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c54e202-d710-4a52-8c33-f773bdd018b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f\" (UID: \"5c54e202-d710-4a52-8c33-f773bdd018b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f" Nov 03 22:33:28.423519 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.423487 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c54e202-d710-4a52-8c33-f773bdd018b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f\" (UID: \"5c54e202-d710-4a52-8c33-f773bdd018b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f" Nov 03 22:33:28.481293 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.481253 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5b4d\" (UniqueName: \"kubernetes.io/projected/5c54e202-d710-4a52-8c33-f773bdd018b5-kube-api-access-p5b4d\") pod \"payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f\" (UID: \"5c54e202-d710-4a52-8c33-f773bdd018b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f" Nov 03 22:33:28.506178 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.506137 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s" Nov 03 22:33:28.602020 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.601914 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw" Nov 03 22:33:28.624592 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.624562 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6462ea50-4efc-40bf-8775-39dd4179bd5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6462ea50-4efc-40bf-8775-39dd4179bd5e-plugins-config-volume\") pod \"6462ea50-4efc-40bf-8775-39dd4179bd5e\" (UID: \"6462ea50-4efc-40bf-8775-39dd4179bd5e\") " Nov 03 22:33:28.624778 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.624643 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6462ea50-4efc-40bf-8775-39dd4179bd5e-kube-api-access-q5nkc\" (UniqueName: \"kubernetes.io/projected/6462ea50-4efc-40bf-8775-39dd4179bd5e-kube-api-access-q5nkc\") pod \"6462ea50-4efc-40bf-8775-39dd4179bd5e\" (UID: \"6462ea50-4efc-40bf-8775-39dd4179bd5e\") " Nov 03 22:33:28.625439 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.625015 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6462ea50-4efc-40bf-8775-39dd4179bd5e-plugins-config-volume" pod "6462ea50-4efc-40bf-8775-39dd4179bd5e" (UID: "6462ea50-4efc-40bf-8775-39dd4179bd5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:28.625439 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.625324 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6462ea50-4efc-40bf-8775-39dd4179bd5e-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:28.627829 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.627795 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6462ea50-4efc-40bf-8775-39dd4179bd5e-kube-api-access-q5nkc" pod "6462ea50-4efc-40bf-8775-39dd4179bd5e" (UID: "6462ea50-4efc-40bf-8775-39dd4179bd5e"). InnerVolumeSpecName "kube-api-access-q5nkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:28.632803 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.632764 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2" path="/var/lib/kubelet/pods/caa2e2bb-c96c-4fd5-9dce-c82ef9d333c2/volumes" Nov 03 22:33:28.687649 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.687294 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f" Nov 03 22:33:28.727161 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.727122 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5nkc\" (UniqueName: \"kubernetes.io/projected/6462ea50-4efc-40bf-8775-39dd4179bd5e-kube-api-access-q5nkc\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:28.993242 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:33:28.993206 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa113bb_18e1_4bd6_9c24_65e16319bf5a.slice/crio-e30f2d85144e22b7d68db724735ff5a9043613d617976ce60884799ab1b8d553 WatchSource:0}: Error finding container e30f2d85144e22b7d68db724735ff5a9043613d617976ce60884799ab1b8d553: Status 404 returned error can't find the container with id e30f2d85144e22b7d68db724735ff5a9043613d617976ce60884799ab1b8d553 Nov 03 22:33:28.993789 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:28.993758 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw"] Nov 03 22:33:29.095967 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:29.095929 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f"] Nov 03 22:33:29.096236 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:33:29.096210 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c54e202_d710_4a52_8c33_f773bdd018b5.slice/crio-d35707dddb597bcc50e9eefe01ef04904a041f3f92a71604645271025895cc97 WatchSource:0}: Error finding container d35707dddb597bcc50e9eefe01ef04904a041f3f92a71604645271025895cc97: Status 404 returned error can't find the container with id d35707dddb597bcc50e9eefe01ef04904a041f3f92a71604645271025895cc97 Nov 03 22:33:29.425927 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:29.425884 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw" event={"ID":"eaa113bb-18e1-4bd6-9c24-65e16319bf5a","Type":"ContainerStarted","Data":"8ab977e10514a64d068fb455dc0c589bdb7999ced18a56fecaeec4920de91148"} Nov 03 22:33:29.426216 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:29.425960 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw" event={"ID":"eaa113bb-18e1-4bd6-9c24-65e16319bf5a","Type":"ContainerStarted","Data":"e30f2d85144e22b7d68db724735ff5a9043613d617976ce60884799ab1b8d553"} Nov 03 22:33:29.426216 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:29.425992 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw" Nov 03 22:33:29.428009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:29.427986 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s" Nov 03 22:33:29.428212 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:29.427982 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s" event={"ID":"6462ea50-4efc-40bf-8775-39dd4179bd5e","Type":"ContainerDied","Data":"ef6a647ba4f53c788202cbedd201e7c65436a15fe3cee62b1f0d19544e4020ea"} Nov 03 22:33:29.428212 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:29.428135 5084 scope.go:122] "RemoveContainer" containerID="0c21a6c6bf2a5ef7577233753892d3ab46a6c68b83020e4e5e71d50e9a2ac70a" Nov 03 22:33:29.430864 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:29.430794 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f" event={"ID":"5c54e202-d710-4a52-8c33-f773bdd018b5","Type":"ContainerStarted","Data":"080d1f4d0f7f4ba4bae6d6f045d772f52e476a047d8c2f0243aa6fcf90f996a8"} Nov 03 22:33:29.431014 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:29.430909 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f" event={"ID":"5c54e202-d710-4a52-8c33-f773bdd018b5","Type":"ContainerStarted","Data":"d35707dddb597bcc50e9eefe01ef04904a041f3f92a71604645271025895cc97"} Nov 03 22:33:29.431813 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:29.431787 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f" Nov 03 22:33:29.545487 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:29.545428 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw" podStartSLOduration=2.545408741 podStartE2EDuration="2.545408741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:29.540314638 +0000 UTC m=+1577.134346614" watchObservedRunningTime="2026-11-03 22:33:29.545408741 +0000 UTC m=+1577.139440715" Nov 03 22:33:29.715118 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:29.714971 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f" podStartSLOduration=2.714946598 podStartE2EDuration="2.714946598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:29.659879493 +0000 UTC m=+1577.253911499" watchObservedRunningTime="2026-11-03 22:33:29.714946598 +0000 UTC m=+1577.308978572" Nov 03 22:33:29.804664 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:29.804619 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s"] Nov 03 22:33:29.809435 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:29.809395 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e1c72fb7-cd7984b78-mnf2s"] Nov 03 22:33:30.468430 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.468362 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v"] Nov 03 22:33:30.470387 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.470353 5084 cpu_manager.go:414] "containerMap: removing container" podUID="6462ea50-4efc-40bf-8775-39dd4179bd5e" containerName="payload-processing" Nov 03 22:33:30.470618 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.470599 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6462ea50-4efc-40bf-8775-39dd4179bd5e" containerName="payload-processing" Nov 03 22:33:30.470938 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.470907 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="6462ea50-4efc-40bf-8775-39dd4179bd5e" containerName="payload-processing" Nov 03 22:33:30.477800 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.477759 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v" Nov 03 22:33:30.504466 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.504413 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx69k\" (UniqueName: \"kubernetes.io/projected/fae1546c-f272-4c56-8d56-5933d34c703b-kube-api-access-cx69k\") pod \"payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v\" (UID: \"fae1546c-f272-4c56-8d56-5933d34c703b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v" Nov 03 22:33:30.504786 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.504761 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae1546c-f272-4c56-8d56-5933d34c703b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v\" (UID: \"fae1546c-f272-4c56-8d56-5933d34c703b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v" Nov 03 22:33:30.546283 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.546246 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-bc88eb47\"" Nov 03 22:33:30.702515 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.616759 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx69k\" (UniqueName: \"kubernetes.io/projected/fae1546c-f272-4c56-8d56-5933d34c703b-kube-api-access-cx69k\") pod \"payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v\" (UID: \"fae1546c-f272-4c56-8d56-5933d34c703b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v" Nov 03 22:33:30.702515 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.616842 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae1546c-f272-4c56-8d56-5933d34c703b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v\" (UID: \"fae1546c-f272-4c56-8d56-5933d34c703b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v" Nov 03 22:33:30.702515 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.618036 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae1546c-f272-4c56-8d56-5933d34c703b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v\" (UID: \"fae1546c-f272-4c56-8d56-5933d34c703b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v" Nov 03 22:33:30.739034 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.719249 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx69k\" (UniqueName: \"kubernetes.io/projected/fae1546c-f272-4c56-8d56-5933d34c703b-kube-api-access-cx69k\") pod \"payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v\" (UID: \"fae1546c-f272-4c56-8d56-5933d34c703b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v" Nov 03 22:33:30.772293 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.769806 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6462ea50-4efc-40bf-8775-39dd4179bd5e" path="/var/lib/kubelet/pods/6462ea50-4efc-40bf-8775-39dd4179bd5e/volumes" Nov 03 22:33:30.772293 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.770336 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v"] Nov 03 22:33:30.772293 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.770402 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4"] Nov 03 22:33:30.792084 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.784474 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v" Nov 03 22:33:30.825472 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.825433 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4"] Nov 03 22:33:30.825827 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.825811 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4" Nov 03 22:33:31.001320 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.993280 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd4b6d3-6913-4191-ac28-028701a386e8-plugins-config-volume\") pod \"payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4\" (UID: \"ccd4b6d3-6913-4191-ac28-028701a386e8\") " pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4" Nov 03 22:33:31.001320 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:30.993360 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcpmq\" (UniqueName: \"kubernetes.io/projected/ccd4b6d3-6913-4191-ac28-028701a386e8-kube-api-access-vcpmq\") pod \"payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4\" (UID: \"ccd4b6d3-6913-4191-ac28-028701a386e8\") " pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4" Nov 03 22:33:31.055733 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:31.054258 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-bc88eb47-dockercfg-8wg7l\"" Nov 03 22:33:31.099893 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:31.099837 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd4b6d3-6913-4191-ac28-028701a386e8-plugins-config-volume\") pod \"payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4\" (UID: \"ccd4b6d3-6913-4191-ac28-028701a386e8\") " pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4" Nov 03 22:33:31.100222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:31.100200 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcpmq\" (UniqueName: \"kubernetes.io/projected/ccd4b6d3-6913-4191-ac28-028701a386e8-kube-api-access-vcpmq\") pod \"payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4\" (UID: \"ccd4b6d3-6913-4191-ac28-028701a386e8\") " pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4" Nov 03 22:33:31.101964 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:31.101927 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd4b6d3-6913-4191-ac28-028701a386e8-plugins-config-volume\") pod \"payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4\" (UID: \"ccd4b6d3-6913-4191-ac28-028701a386e8\") " pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4" Nov 03 22:33:31.159042 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:31.158998 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcpmq\" (UniqueName: \"kubernetes.io/projected/ccd4b6d3-6913-4191-ac28-028701a386e8-kube-api-access-vcpmq\") pod \"payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4\" (UID: \"ccd4b6d3-6913-4191-ac28-028701a386e8\") " pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4" Nov 03 22:33:31.398286 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:31.398246 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4" Nov 03 22:33:31.451906 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:31.451855 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v"] Nov 03 22:33:31.453998 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:33:31.453964 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae1546c_f272_4c56_8d56_5933d34c703b.slice/crio-31fbaf2ed606aac9852c0cd018f459a77998370a0957f0aa76045414f0ed1947 WatchSource:0}: Error finding container 31fbaf2ed606aac9852c0cd018f459a77998370a0957f0aa76045414f0ed1947: Status 404 returned error can't find the container with id 31fbaf2ed606aac9852c0cd018f459a77998370a0957f0aa76045414f0ed1947 Nov 03 22:33:31.564821 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:31.564740 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v" event={"ID":"fae1546c-f272-4c56-8d56-5933d34c703b","Type":"ContainerStarted","Data":"31fbaf2ed606aac9852c0cd018f459a77998370a0957f0aa76045414f0ed1947"} Nov 03 22:33:31.795477 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:33:31.795440 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd4b6d3_6913_4191_ac28_028701a386e8.slice/crio-84e6dbfaee1c661dc173a10feb21521ed1bb4e6c12d013a9af32201990ccd6b5 WatchSource:0}: Error finding container 84e6dbfaee1c661dc173a10feb21521ed1bb4e6c12d013a9af32201990ccd6b5: Status 404 returned error can't find the container with id 84e6dbfaee1c661dc173a10feb21521ed1bb4e6c12d013a9af32201990ccd6b5 Nov 03 22:33:31.795646 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:31.795514 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4"] Nov 03 22:33:32.575859 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:32.575818 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4" event={"ID":"ccd4b6d3-6913-4191-ac28-028701a386e8","Type":"ContainerStarted","Data":"0091922fcb609de295e6b47565f9e9d9a350642629a2c13ab4e06d8941c82dcf"} Nov 03 22:33:32.576378 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:32.575887 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4" event={"ID":"ccd4b6d3-6913-4191-ac28-028701a386e8","Type":"ContainerStarted","Data":"84e6dbfaee1c661dc173a10feb21521ed1bb4e6c12d013a9af32201990ccd6b5"} Nov 03 22:33:32.576378 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:32.575917 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4" Nov 03 22:33:32.577894 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:32.577866 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v" event={"ID":"fae1546c-f272-4c56-8d56-5933d34c703b","Type":"ContainerStarted","Data":"cb05c3fffba49143b07146a728c341cfae9b5bd5ef7ef71b7e449b0abc59e443"} Nov 03 22:33:32.578033 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:32.577991 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v" Nov 03 22:33:32.696836 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:32.696763 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4" podStartSLOduration=2.696741797 podStartE2EDuration="2.696741797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:32.692397212 +0000 UTC m=+1580.286429187" watchObservedRunningTime="2026-11-03 22:33:32.696741797 +0000 UTC m=+1580.290773773" Nov 03 22:33:32.898385 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:32.898237 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v" podStartSLOduration=2.898214766 podStartE2EDuration="2.898214766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:32.887560754 +0000 UTC m=+1580.481592729" watchObservedRunningTime="2026-11-03 22:33:32.898214766 +0000 UTC m=+1580.492246742" Nov 03 22:33:40.575352 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:40.575318 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw" Nov 03 22:33:41.569809 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:41.569776 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f" Nov 03 22:33:42.024882 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:33:42.024831 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:33:42.509626 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:42.509527 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw"] Nov 03 22:33:42.509803 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:42.509783 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" podUID="e5034590-d620-46de-92ee-22cf08dd39be" containerName="maas-api" containerID="cri-o://5a75ccb86dd938e49281965a2020fd4b1cf74e54de1b91c9b5c693b961b37b7c" gracePeriod=30 Nov 03 22:33:42.699483 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:42.699450 5084 generic.go:356] "Generic (PLEG): container finished" podID="e5034590-d620-46de-92ee-22cf08dd39be" containerID="5a75ccb86dd938e49281965a2020fd4b1cf74e54de1b91c9b5c693b961b37b7c" exitCode=0 Nov 03 22:33:42.699581 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:42.699521 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" event={"ID":"e5034590-d620-46de-92ee-22cf08dd39be","Type":"ContainerDied","Data":"5a75ccb86dd938e49281965a2020fd4b1cf74e54de1b91c9b5c693b961b37b7c"} Nov 03 22:33:43.197523 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:43.186120 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:33:43.202120 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:43.202057 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5034590-d620-46de-92ee-22cf08dd39be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5034590-d620-46de-92ee-22cf08dd39be-metrics-certs\") pod \"e5034590-d620-46de-92ee-22cf08dd39be\" (UID: \"e5034590-d620-46de-92ee-22cf08dd39be\") " Nov 03 22:33:43.203990 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:43.203961 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5034590-d620-46de-92ee-22cf08dd39be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5034590-d620-46de-92ee-22cf08dd39be-maas-api-tls\") pod \"e5034590-d620-46de-92ee-22cf08dd39be\" (UID: \"e5034590-d620-46de-92ee-22cf08dd39be\") " Nov 03 22:33:43.204379 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:43.204352 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5034590-d620-46de-92ee-22cf08dd39be-kube-api-access-xm8pk\" (UniqueName: \"kubernetes.io/projected/e5034590-d620-46de-92ee-22cf08dd39be-kube-api-access-xm8pk\") pod \"e5034590-d620-46de-92ee-22cf08dd39be\" (UID: \"e5034590-d620-46de-92ee-22cf08dd39be\") " Nov 03 22:33:43.205008 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:43.204984 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5034590-d620-46de-92ee-22cf08dd39be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5034590-d620-46de-92ee-22cf08dd39be-openshift-service-ca\") pod \"e5034590-d620-46de-92ee-22cf08dd39be\" (UID: \"e5034590-d620-46de-92ee-22cf08dd39be\") " Nov 03 22:33:43.205732 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:43.205609 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5034590-d620-46de-92ee-22cf08dd39be-openshift-service-ca" pod "e5034590-d620-46de-92ee-22cf08dd39be" (UID: "e5034590-d620-46de-92ee-22cf08dd39be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:43.206922 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:43.206898 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5034590-d620-46de-92ee-22cf08dd39be-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:43.208958 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:43.208927 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5034590-d620-46de-92ee-22cf08dd39be-metrics-certs" pod "e5034590-d620-46de-92ee-22cf08dd39be" (UID: "e5034590-d620-46de-92ee-22cf08dd39be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:33:43.210252 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:43.210202 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5034590-d620-46de-92ee-22cf08dd39be-kube-api-access-xm8pk" pod "e5034590-d620-46de-92ee-22cf08dd39be" (UID: "e5034590-d620-46de-92ee-22cf08dd39be"). InnerVolumeSpecName "kube-api-access-xm8pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:43.211158 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:43.211116 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5034590-d620-46de-92ee-22cf08dd39be-maas-api-tls" pod "e5034590-d620-46de-92ee-22cf08dd39be" (UID: "e5034590-d620-46de-92ee-22cf08dd39be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:33:43.308576 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:43.308531 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5034590-d620-46de-92ee-22cf08dd39be-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:43.308576 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:43.308570 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5034590-d620-46de-92ee-22cf08dd39be-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:43.308576 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:43.308581 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm8pk\" (UniqueName: \"kubernetes.io/projected/e5034590-d620-46de-92ee-22cf08dd39be-kube-api-access-xm8pk\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:33:43.615111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:43.614267 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v" Nov 03 22:33:43.615711 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:43.615682 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4" Nov 03 22:33:43.758665 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:43.758601 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" event={"ID":"e5034590-d620-46de-92ee-22cf08dd39be","Type":"ContainerDied","Data":"0187afd19c6abf8a1c458793cb9d43bc9b4dccbd370ad9ed763a8bdab7082942"} Nov 03 22:33:43.758870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:43.758702 5084 scope.go:122] "RemoveContainer" containerID="5a75ccb86dd938e49281965a2020fd4b1cf74e54de1b91c9b5c693b961b37b7c" Nov 03 22:33:43.758973 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:43.758955 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw" Nov 03 22:33:44.005851 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:44.005753 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw"] Nov 03 22:33:44.010244 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:44.010206 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1c72fb7-5599c5964f-wntnw"] Nov 03 22:33:44.682088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:44.679582 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5034590-d620-46de-92ee-22cf08dd39be" path="/var/lib/kubelet/pods/e5034590-d620-46de-92ee-22cf08dd39be/volumes" Nov 03 22:33:54.104086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:54.103976 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6"] Nov 03 22:33:54.156150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:54.155050 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84bfa22-2h2p6"] Nov 03 22:33:54.629779 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:54.629738 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34" path="/var/lib/kubelet/pods/4ec16b4a-f5d0-4ced-ba33-3fc1ca1b9c34/volumes" Nov 03 22:33:58.249166 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.249126 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh"] Nov 03 22:33:58.249929 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.249910 5084 cpu_manager.go:414] "containerMap: removing container" podUID="e5034590-d620-46de-92ee-22cf08dd39be" containerName="maas-api" Nov 03 22:33:58.249929 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.249928 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5034590-d620-46de-92ee-22cf08dd39be" containerName="maas-api" Nov 03 22:33:58.250123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.250112 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5034590-d620-46de-92ee-22cf08dd39be" containerName="maas-api" Nov 03 22:33:58.255399 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.255355 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:33:58.310748 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.310713 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh"] Nov 03 22:33:58.330021 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.329971 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjz8p\" (UniqueName: \"kubernetes.io/projected/59460ed3-3f95-4092-abe1-a42780972f29-kube-api-access-tjz8p\") pod \"maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh\" (UID: \"59460ed3-3f95-4092-abe1-a42780972f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:33:58.330218 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.330155 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59460ed3-3f95-4092-abe1-a42780972f29-openshift-service-ca\") pod \"maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh\" (UID: \"59460ed3-3f95-4092-abe1-a42780972f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:33:58.330294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.330278 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59460ed3-3f95-4092-abe1-a42780972f29-metrics-certs\") pod \"maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh\" (UID: \"59460ed3-3f95-4092-abe1-a42780972f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:33:58.330380 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.330352 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59460ed3-3f95-4092-abe1-a42780972f29-maas-api-tls\") pod \"maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh\" (UID: \"59460ed3-3f95-4092-abe1-a42780972f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:33:58.358655 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.358606 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c211881d-serving-cert\"" Nov 03 22:33:58.431858 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.431816 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59460ed3-3f95-4092-abe1-a42780972f29-maas-api-tls\") pod \"maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh\" (UID: \"59460ed3-3f95-4092-abe1-a42780972f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:33:58.432090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.431928 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjz8p\" (UniqueName: \"kubernetes.io/projected/59460ed3-3f95-4092-abe1-a42780972f29-kube-api-access-tjz8p\") pod \"maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh\" (UID: \"59460ed3-3f95-4092-abe1-a42780972f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:33:58.432090 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:33:58.431963 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-serving-cert: secret "maas-api-e2e-ait-c211881d-serving-cert" not found Nov 03 22:33:58.432090 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:33:58.432027 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59460ed3-3f95-4092-abe1-a42780972f29-maas-api-tls podName:59460ed3-3f95-4092-abe1-a42780972f29 nodeName:}" failed. No retries permitted until 2026-11-03 22:33:58.932007887 +0000 UTC m=+1606.526039838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59460ed3-3f95-4092-abe1-a42780972f29-maas-api-tls") pod "maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" (UID: "59460ed3-3f95-4092-abe1-a42780972f29") : secret "maas-api-e2e-ait-c211881d-serving-cert" not found Nov 03 22:33:58.432265 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.432107 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59460ed3-3f95-4092-abe1-a42780972f29-openshift-service-ca\") pod \"maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh\" (UID: \"59460ed3-3f95-4092-abe1-a42780972f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:33:58.432265 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.432200 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59460ed3-3f95-4092-abe1-a42780972f29-metrics-certs\") pod \"maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh\" (UID: \"59460ed3-3f95-4092-abe1-a42780972f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:33:58.432953 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.432915 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59460ed3-3f95-4092-abe1-a42780972f29-openshift-service-ca\") pod \"maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh\" (UID: \"59460ed3-3f95-4092-abe1-a42780972f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:33:58.439988 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.439952 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59460ed3-3f95-4092-abe1-a42780972f29-metrics-certs\") pod \"maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh\" (UID: \"59460ed3-3f95-4092-abe1-a42780972f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:33:58.517647 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.517610 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjz8p\" (UniqueName: \"kubernetes.io/projected/59460ed3-3f95-4092-abe1-a42780972f29-kube-api-access-tjz8p\") pod \"maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh\" (UID: \"59460ed3-3f95-4092-abe1-a42780972f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:33:58.955159 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.954995 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59460ed3-3f95-4092-abe1-a42780972f29-maas-api-tls\") pod \"maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh\" (UID: \"59460ed3-3f95-4092-abe1-a42780972f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:33:58.957978 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:58.957938 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59460ed3-3f95-4092-abe1-a42780972f29-maas-api-tls\") pod \"maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh\" (UID: \"59460ed3-3f95-4092-abe1-a42780972f29\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:33:59.177922 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:59.177881 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:33:59.574715 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:33:59.574668 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh"] Nov 03 22:33:59.575969 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:33:59.575942 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59460ed3_3f95_4092_abe1_a42780972f29.slice/crio-e504930597bf831b83a7dea0c8b92fb3220869b32a1bd48ff54293c14cdd4cea WatchSource:0}: Error finding container e504930597bf831b83a7dea0c8b92fb3220869b32a1bd48ff54293c14cdd4cea: Status 404 returned error can't find the container with id e504930597bf831b83a7dea0c8b92fb3220869b32a1bd48ff54293c14cdd4cea Nov 03 22:34:00.032228 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:00.032182 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" event={"ID":"59460ed3-3f95-4092-abe1-a42780972f29","Type":"ContainerStarted","Data":"e504930597bf831b83a7dea0c8b92fb3220869b32a1bd48ff54293c14cdd4cea"} Nov 03 22:34:01.042375 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:01.042333 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" event={"ID":"59460ed3-3f95-4092-abe1-a42780972f29","Type":"ContainerStarted","Data":"cb3ecf5cbf111abfd5ae9a5f62c762065919f7b47a3a16bd7afef18f59b92e21"} Nov 03 22:34:01.042811 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:01.042409 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:34:01.161834 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:01.161764 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" podStartSLOduration=2.187087316 podStartE2EDuration="3.161747157s" totalImagesPullingTime="974.659841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:01.159614961 +0000 UTC m=+1608.753646935" watchObservedRunningTime="2026-11-03 22:34:01.161747157 +0000 UTC m=+1608.755779150" Nov 03 22:34:07.057122 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:07.057057 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:34:07.545402 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:07.545361 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq"] Nov 03 22:34:07.551246 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:07.551205 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" Nov 03 22:34:07.602852 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:07.602810 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq"] Nov 03 22:34:07.604291 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:07.604251 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-bc88eb47-serving-cert\"" Nov 03 22:34:07.717315 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:07.717267 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-openshift-service-ca\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq\" (UID: \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" Nov 03 22:34:07.717534 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:07.717413 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-metrics-certs\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq\" (UID: \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" Nov 03 22:34:07.717652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:07.717542 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj468\" (UniqueName: \"kubernetes.io/projected/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-kube-api-access-dj468\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq\" (UID: \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" Nov 03 22:34:07.717708 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:07.717685 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-maas-api-tls\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq\" (UID: \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" Nov 03 22:34:07.825038 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:07.819392 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-openshift-service-ca\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq\" (UID: \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" Nov 03 22:34:07.825038 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:07.819538 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-metrics-certs\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq\" (UID: \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" Nov 03 22:34:07.825038 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:07.819647 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj468\" (UniqueName: \"kubernetes.io/projected/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-kube-api-access-dj468\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq\" (UID: \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" Nov 03 22:34:07.825038 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:07.819863 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-maas-api-tls\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq\" (UID: \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" Nov 03 22:34:07.834141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:07.831745 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-openshift-service-ca\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq\" (UID: \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" Nov 03 22:34:07.861249 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:07.861212 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-metrics-certs\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq\" (UID: \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" Nov 03 22:34:07.873261 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:07.873220 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-maas-api-tls\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq\" (UID: \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" Nov 03 22:34:07.906820 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:07.906780 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj468\" (UniqueName: \"kubernetes.io/projected/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-kube-api-access-dj468\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq\" (UID: \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" Nov 03 22:34:08.167436 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:08.167324 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" Nov 03 22:34:08.561311 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:08.559704 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq"] Nov 03 22:34:08.562468 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:34:08.562433 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4872ed1e_82e9_412c_a5c5_1ae3a4cf4a61.slice/crio-daa6e344030d472dac56c90ff71e0ad6f41da40fc7274cdef56ba2fe8e4f4420 WatchSource:0}: Error finding container daa6e344030d472dac56c90ff71e0ad6f41da40fc7274cdef56ba2fe8e4f4420: Status 404 returned error can't find the container with id daa6e344030d472dac56c90ff71e0ad6f41da40fc7274cdef56ba2fe8e4f4420 Nov 03 22:34:09.123321 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:09.123280 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" event={"ID":"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61","Type":"ContainerStarted","Data":"daa6e344030d472dac56c90ff71e0ad6f41da40fc7274cdef56ba2fe8e4f4420"} Nov 03 22:34:10.134163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:10.134121 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" event={"ID":"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61","Type":"ContainerStarted","Data":"8c440dda66ef41bdfbded79f3fc4e35791bc1a32590d846063519dc39ba50100"} Nov 03 22:34:10.134659 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:10.134241 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" Nov 03 22:34:10.254195 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:10.254127 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" podStartSLOduration=2.375359734 podStartE2EDuration="3.254102557s" totalImagesPullingTime="878.742823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:10.250492525 +0000 UTC m=+1617.844524503" watchObservedRunningTime="2026-11-03 22:34:10.254102557 +0000 UTC m=+1617.848134533" Nov 03 22:34:12.026315 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:34:12.026276 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:34:12.048938 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:12.048901 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh"] Nov 03 22:34:12.100861 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:12.100819 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49a58bb0-t2hnh"] Nov 03 22:34:12.652133 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:12.652088 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d997c646-e8fd-4a08-99b6-603ab21587a7" path="/var/lib/kubelet/pods/d997c646-e8fd-4a08-99b6-603ab21587a7/volumes" Nov 03 22:34:14.051229 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.051188 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh"] Nov 03 22:34:14.102554 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.102512 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e9fef24-xsgwh"] Nov 03 22:34:14.274503 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.274458 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj"] Nov 03 22:34:14.280680 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.280648 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.332872 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.332779 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj"] Nov 03 22:34:14.362547 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.362502 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d5085e-0b4b-46a0-83cc-339d5550dd95-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.362738 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.362715 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3d5085e-0b4b-46a0-83cc-339d5550dd95-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.362791 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.362776 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3d5085e-0b4b-46a0-83cc-339d5550dd95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.362934 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.362908 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d5085e-0b4b-46a0-83cc-339d5550dd95-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.362994 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.362950 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3d5085e-0b4b-46a0-83cc-339d5550dd95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.363056 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.363036 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3d5085e-0b4b-46a0-83cc-339d5550dd95-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.363179 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.363124 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3d5085e-0b4b-46a0-83cc-339d5550dd95-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.363236 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.363196 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3d5085e-0b4b-46a0-83cc-339d5550dd95-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.363290 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.363277 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltpsm\" (UniqueName: \"kubernetes.io/projected/d3d5085e-0b4b-46a0-83cc-339d5550dd95-kube-api-access-ltpsm\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.378609 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.378565 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kf4rv\"" Nov 03 22:34:14.465334 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.465287 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d5085e-0b4b-46a0-83cc-339d5550dd95-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.465529 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.465438 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3d5085e-0b4b-46a0-83cc-339d5550dd95-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.465529 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.465488 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3d5085e-0b4b-46a0-83cc-339d5550dd95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.465644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.465561 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d5085e-0b4b-46a0-83cc-339d5550dd95-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.465644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.465585 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3d5085e-0b4b-46a0-83cc-339d5550dd95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.465644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.465619 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3d5085e-0b4b-46a0-83cc-339d5550dd95-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.465819 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.465657 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3d5085e-0b4b-46a0-83cc-339d5550dd95-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.465819 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.465704 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3d5085e-0b4b-46a0-83cc-339d5550dd95-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.465819 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.465731 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d5085e-0b4b-46a0-83cc-339d5550dd95-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.465988 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.465818 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltpsm\" (UniqueName: \"kubernetes.io/projected/d3d5085e-0b4b-46a0-83cc-339d5550dd95-kube-api-access-ltpsm\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.465988 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.465933 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d5085e-0b4b-46a0-83cc-339d5550dd95-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.466350 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.466326 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3d5085e-0b4b-46a0-83cc-339d5550dd95-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.466456 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.466361 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3d5085e-0b4b-46a0-83cc-339d5550dd95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.466456 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.466369 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3d5085e-0b4b-46a0-83cc-339d5550dd95-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.467905 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.467874 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3d5085e-0b4b-46a0-83cc-339d5550dd95-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.468219 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.468195 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3d5085e-0b4b-46a0-83cc-339d5550dd95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.523961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.523921 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3d5085e-0b4b-46a0-83cc-339d5550dd95-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.524150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.524023 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltpsm\" (UniqueName: \"kubernetes.io/projected/d3d5085e-0b4b-46a0-83cc-339d5550dd95-kube-api-access-ltpsm\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-28lxj\" (UID: \"d3d5085e-0b4b-46a0-83cc-339d5550dd95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.598981 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.598886 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:14.632150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:14.632114 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb779fef-d86b-4b16-ad9f-78459edd4e98" path="/var/lib/kubelet/pods/cb779fef-d86b-4b16-ad9f-78459edd4e98/volumes" Nov 03 22:34:15.043461 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:15.043427 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj"] Nov 03 22:34:15.043694 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:34:15.043666 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d5085e_0b4b_46a0_83cc_339d5550dd95.slice/crio-bc05d427f7bbde5afd85e83b9adb2f798ca5fdda0652899d8ae0d03bbfc0e3fd WatchSource:0}: Error finding container bc05d427f7bbde5afd85e83b9adb2f798ca5fdda0652899d8ae0d03bbfc0e3fd: Status 404 returned error can't find the container with id bc05d427f7bbde5afd85e83b9adb2f798ca5fdda0652899d8ae0d03bbfc0e3fd Nov 03 22:34:15.046446 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:15.046407 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:34:15.046562 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:15.046480 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:34:15.046562 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:15.046523 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:34:15.182444 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:15.182405 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" event={"ID":"d3d5085e-0b4b-46a0-83cc-339d5550dd95","Type":"ContainerStarted","Data":"5b25cb056d6fecda9d149e1add5b152b7c8eabc924cbee580cd1cfd9eec240c2"} Nov 03 22:34:15.183052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:15.182470 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" event={"ID":"d3d5085e-0b4b-46a0-83cc-339d5550dd95","Type":"ContainerStarted","Data":"bc05d427f7bbde5afd85e83b9adb2f798ca5fdda0652899d8ae0d03bbfc0e3fd"} Nov 03 22:34:15.304347 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:15.304226 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" podStartSLOduration=1.304201781 podStartE2EDuration="1.304201781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:15.300272473 +0000 UTC m=+1622.894304449" watchObservedRunningTime="2026-11-03 22:34:15.304201781 +0000 UTC m=+1622.898233756" Nov 03 22:34:15.599477 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:15.599377 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:15.789954 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:15.789903 5084 scope.go:122] "RemoveContainer" containerID="45a77e9fe20b49bcad9801ba972560bcb684924f940d7a86b2ece6e65500b05f" Nov 03 22:34:16.150824 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:16.150790 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" Nov 03 22:34:16.602002 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:16.601961 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-28lxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:34:16.602460 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:16.602029 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" podUID="d3d5085e-0b4b-46a0-83cc-339d5550dd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:34:17.600052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:17.600003 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-28lxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:34:17.600295 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:17.600093 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" podUID="d3d5085e-0b4b-46a0-83cc-339d5550dd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:34:18.599730 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:18.599690 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-28lxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:34:18.600289 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:18.599754 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" podUID="d3d5085e-0b4b-46a0-83cc-339d5550dd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:34:19.599746 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:19.599702 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-28lxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:34:19.600161 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:19.599774 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" podUID="d3d5085e-0b4b-46a0-83cc-339d5550dd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:34:20.599736 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:20.599693 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-28lxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:34:20.599936 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:20.599762 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" podUID="d3d5085e-0b4b-46a0-83cc-339d5550dd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:34:21.600055 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:21.600016 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-28lxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:34:21.600488 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:21.600112 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" podUID="d3d5085e-0b4b-46a0-83cc-339d5550dd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:34:22.599657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:22.599615 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-28lxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 22:34:22.599867 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:22.599694 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" podUID="d3d5085e-0b4b-46a0-83cc-339d5550dd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 22:34:23.987302 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:23.987264 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:23.987827 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:23.987799 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:23.988907 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:23.988880 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" Nov 03 22:34:24.097207 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:24.097161 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd"] Nov 03 22:34:24.101390 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:24.101359 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8f0bd85-7t7bd"] Nov 03 22:34:24.630992 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:24.630952 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85c67b7-19ff-4981-8f38-faac63eed31d" path="/var/lib/kubelet/pods/d85c67b7-19ff-4981-8f38-faac63eed31d/volumes" Nov 03 22:34:29.057091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:29.057020 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk"] Nov 03 22:34:29.107233 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:29.107193 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9527ca05-8lkqk"] Nov 03 22:34:30.631133 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:30.631089 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa23f164-97be-463e-9dd1-1dbae8a0b59c" path="/var/lib/kubelet/pods/fa23f164-97be-463e-9dd1-1dbae8a0b59c/volumes" Nov 03 22:34:42.029305 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:34:42.029264 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:34:46.049653 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:46.049615 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts"] Nov 03 22:34:46.102960 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:46.102919 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c5a1324-6cvts"] Nov 03 22:34:46.630502 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:46.630464 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="865851d3-e8f5-40de-8770-34ecfafdf219" path="/var/lib/kubelet/pods/865851d3-e8f5-40de-8770-34ecfafdf219/volumes" Nov 03 22:34:46.806966 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:46.806922 5084 generic.go:356] "Generic (PLEG): container finished" podID="18e734ea-1f54-4603-9fd9-3261ba6d4ec7" containerID="a5d01c285a29b9b9289e19676165eafd86047db93daacdc38be306402a49aaba" exitCode=0 Nov 03 22:34:46.807215 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:46.806994 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" event={"ID":"18e734ea-1f54-4603-9fd9-3261ba6d4ec7","Type":"ContainerDied","Data":"a5d01c285a29b9b9289e19676165eafd86047db93daacdc38be306402a49aaba"} Nov 03 22:34:46.807557 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:46.807540 5084 scope.go:122] "RemoveContainer" containerID="a5d01c285a29b9b9289e19676165eafd86047db93daacdc38be306402a49aaba" Nov 03 22:34:46.809253 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:46.809230 5084 generic.go:356] "Generic (PLEG): container finished" podID="ad18cacf-d8cc-4b0d-909e-7c60805abe2f" containerID="882ba25e5635fb50655d1c4f91fda96eae73ffe1b16a2731aaccc522a217aa93" exitCode=0 Nov 03 22:34:46.809373 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:46.809304 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" event={"ID":"ad18cacf-d8cc-4b0d-909e-7c60805abe2f","Type":"ContainerDied","Data":"882ba25e5635fb50655d1c4f91fda96eae73ffe1b16a2731aaccc522a217aa93"} Nov 03 22:34:46.809521 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:46.809487 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:34:46.809593 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:46.809566 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:34:46.809648 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:46.809607 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:34:46.809794 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:46.809776 5084 scope.go:122] "RemoveContainer" containerID="882ba25e5635fb50655d1c4f91fda96eae73ffe1b16a2731aaccc522a217aa93" Nov 03 22:34:46.811247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:46.811219 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:34:46.811343 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:46.811302 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:34:46.811407 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:46.811355 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:34:47.840301 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:47.840258 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" event={"ID":"ad18cacf-d8cc-4b0d-909e-7c60805abe2f","Type":"ContainerStarted","Data":"c109be0fb54c3468b9277ae28aedd0104fb9e2458f30cce82c2f32d1932edd3c"} Nov 03 22:34:47.842523 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:47.842494 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" event={"ID":"18e734ea-1f54-4603-9fd9-3261ba6d4ec7","Type":"ContainerStarted","Data":"751113cfbd3567b732578d94df98a04dd3c74c25941308cf9d2bc41152e1f18c"} Nov 03 22:34:48.217190 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:48.213513 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:34:48.218776 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:48.218733 5084 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:34:48.219045 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:48.219015 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" podUID="ad18cacf-d8cc-4b0d-909e-7c60805abe2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 22:34:48.568647 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:48.568599 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:34:48.569028 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:48.569005 5084 patch_prober.go:29] interesting pod/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 22:34:48.569235 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:48.569208 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" podUID="18e734ea-1f54-4603-9fd9-3261ba6d4ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 22:34:49.148957 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:49.148232 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:34:49.212087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:49.209516 5084 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:34:49.212087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:49.209585 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" podUID="ad18cacf-d8cc-4b0d-909e-7c60805abe2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 22:34:49.569512 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:49.569472 5084 patch_prober.go:29] interesting pod/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 22:34:49.569836 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:49.569803 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" podUID="18e734ea-1f54-4603-9fd9-3261ba6d4ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 22:34:50.165087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:50.160176 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:34:50.210636 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:50.210589 5084 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:34:50.210923 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:50.210892 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" podUID="ad18cacf-d8cc-4b0d-909e-7c60805abe2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 22:34:50.579089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:50.578207 5084 patch_prober.go:29] interesting pod/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 22:34:50.579089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:50.578275 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" podUID="18e734ea-1f54-4603-9fd9-3261ba6d4ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 22:34:51.048553 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:51.048508 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg"] Nov 03 22:34:51.112083 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:51.112009 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-950f43c1-x65wg"] Nov 03 22:34:51.208720 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:51.208676 5084 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:34:51.209443 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:51.209391 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" podUID="ad18cacf-d8cc-4b0d-909e-7c60805abe2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 22:34:51.570447 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:51.570407 5084 patch_prober.go:29] interesting pod/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 22:34:51.570766 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:51.570729 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" podUID="18e734ea-1f54-4603-9fd9-3261ba6d4ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 22:34:52.211092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:52.208534 5084 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:34:52.211092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:52.208623 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" podUID="ad18cacf-d8cc-4b0d-909e-7c60805abe2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 22:34:52.568580 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:52.568539 5084 patch_prober.go:29] interesting pod/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 22:34:52.568962 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:52.568927 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" podUID="18e734ea-1f54-4603-9fd9-3261ba6d4ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 22:34:52.630159 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:52.630125 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70844697-f6c3-4f12-8fe3-c0d80d501784" path="/var/lib/kubelet/pods/70844697-f6c3-4f12-8fe3-c0d80d501784/volumes" Nov 03 22:34:53.212932 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:53.212877 5084 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 22:34:53.222017 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:53.221960 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" podUID="ad18cacf-d8cc-4b0d-909e-7c60805abe2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 22:34:53.565570 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:53.565528 5084 patch_prober.go:29] interesting pod/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 22:34:53.565918 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:53.565882 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" podUID="18e734ea-1f54-4603-9fd9-3261ba6d4ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 22:34:54.213305 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:54.213269 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:34:54.214495 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:54.214470 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:34:54.571277 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:54.571241 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:34:54.573933 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:54.573894 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:34:56.057217 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:56.057177 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k"] Nov 03 22:34:56.108023 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:56.107971 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fee5eb7a-bb95k"] Nov 03 22:34:56.630341 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:56.630301 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6665a28-c649-487e-ab34-868ed000c53c" path="/var/lib/kubelet/pods/b6665a28-c649-487e-ab34-868ed000c53c/volumes" Nov 03 22:34:59.647731 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.647688 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9"] Nov 03 22:34:59.656492 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.656449 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.739460 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.739424 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9"] Nov 03 22:34:59.808173 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.808126 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.808386 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.808186 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.808386 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.808292 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.808386 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.808366 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.808557 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.808395 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.808557 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.808504 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.808659 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.808567 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r5b7\" (UniqueName: \"kubernetes.io/projected/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-kube-api-access-8r5b7\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.808659 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.808651 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.808766 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.808684 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.909936 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.909830 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.909936 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.909884 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.910211 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.909964 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.910211 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.910013 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.910211 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.910128 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.910211 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.910191 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.910425 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.910230 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.910425 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.910296 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.910425 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.910392 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r5b7\" (UniqueName: \"kubernetes.io/projected/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-kube-api-access-8r5b7\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.910586 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.910454 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.911030 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.910674 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.911030 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.910745 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.911030 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.910895 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.911030 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.910910 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.913108 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.913056 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.913353 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.913333 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.968903 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.968861 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.969078 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.968862 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r5b7\" (UniqueName: \"kubernetes.io/projected/d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2-kube-api-access-8r5b7\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-l96p9\" (UID: \"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:34:59.986209 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:34:59.986168 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:35:00.454428 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:35:00.454385 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75afbc1_4de4_4b68_ae28_e0a8c0d39fe2.slice/crio-47338cdc086fb10ef434faee393aebe0c704faf3b8c3e4ce6948c8bb8f1631d3 WatchSource:0}: Error finding container 47338cdc086fb10ef434faee393aebe0c704faf3b8c3e4ce6948c8bb8f1631d3: Status 404 returned error can't find the container with id 47338cdc086fb10ef434faee393aebe0c704faf3b8c3e4ce6948c8bb8f1631d3 Nov 03 22:35:00.454742 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:00.454718 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9"] Nov 03 22:35:00.457553 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:00.457506 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:35:00.457676 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:00.457585 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:35:00.457676 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:00.457635 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:35:01.041633 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:01.041587 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" event={"ID":"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2","Type":"ContainerStarted","Data":"0640deb051b99b6fafadc2deb2ece91e959d605531387675cab7d6ddf99cd5e1"} Nov 03 22:35:01.042118 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:01.041659 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" event={"ID":"d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2","Type":"ContainerStarted","Data":"47338cdc086fb10ef434faee393aebe0c704faf3b8c3e4ce6948c8bb8f1631d3"} Nov 03 22:35:01.173655 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:01.173578 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" podStartSLOduration=2.173552667 podStartE2EDuration="2.173552667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:01.162626972 +0000 UTC m=+1668.756658971" watchObservedRunningTime="2026-11-03 22:35:01.173552667 +0000 UTC m=+1668.767584633" Nov 03 22:35:01.986444 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:01.986393 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:35:01.988038 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:01.988003 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-l96p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:35:01.988224 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:01.988073 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" podUID="d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:35:02.986792 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:02.986746 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-l96p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:35:02.987428 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:02.986823 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" podUID="d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:35:03.990013 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:03.989967 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-l96p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:35:03.990689 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:03.990653 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" podUID="d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:35:04.223582 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:04.223541 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw"] Nov 03 22:35:04.349586 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:04.349548 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f"] Nov 03 22:35:04.991414 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:04.988536 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-l96p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:35:04.991414 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:04.988607 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" podUID="d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:35:05.986760 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:05.986715 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-l96p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:35:05.987148 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:05.987107 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" podUID="d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:35:06.495666 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.495616 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt"] Nov 03 22:35:06.501840 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.501804 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.598433 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.598385 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-37b70b-openshift-default-dockercfg-cgtgd\"" Nov 03 22:35:06.613118 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.610640 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt"] Nov 03 22:35:06.624224 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.624164 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-credential-socket\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.624224 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.624223 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-data\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.624497 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.624261 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-podinfo\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.624497 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.624316 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-envoy\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.624497 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.624433 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-token\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.624666 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.624514 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istiod-ca-cert\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.624666 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.624546 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-workload-certs\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.624666 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.624576 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vntb9\" (UniqueName: \"kubernetes.io/projected/9477ee77-a2d4-4500-9bf8-5f2b8f363928-kube-api-access-vntb9\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.624666 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.624635 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-workload-socket\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.777150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.729033 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-token\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.777150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.729172 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istiod-ca-cert\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.777150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.729208 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-workload-certs\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.777150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.729241 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vntb9\" (UniqueName: \"kubernetes.io/projected/9477ee77-a2d4-4500-9bf8-5f2b8f363928-kube-api-access-vntb9\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.777150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.729302 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-workload-socket\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.777150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.729487 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-credential-socket\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.777150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.729513 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-data\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.777150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.729548 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-podinfo\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.777150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.729604 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-envoy\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.777150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.767655 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-workload-socket\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.777150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.767930 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-credential-socket\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.777150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.768284 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-data\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.798092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.779776 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istiod-ca-cert\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:06.988677 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:06.799530 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-workload-certs\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:07.018096 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:07.005885 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-envoy\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:07.018096 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:07.007547 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-l96p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:35:07.018096 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:07.007607 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" podUID="d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:35:07.102167 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:07.076631 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-podinfo\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:07.113852 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:07.113785 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-token\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:07.120152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:07.118726 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vntb9\" (UniqueName: \"kubernetes.io/projected/9477ee77-a2d4-4500-9bf8-5f2b8f363928-kube-api-access-vntb9\") pod \"e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:07.268324 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:07.268282 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:07.752209 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:07.752171 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt"] Nov 03 22:35:07.752907 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:35:07.752874 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9477ee77_a2d4_4500_9bf8_5f2b8f363928.slice/crio-2fb54e46be2b0bef792ef5c1c5664e63ce3647ca41c9000f22db7ac97271271d WatchSource:0}: Error finding container 2fb54e46be2b0bef792ef5c1c5664e63ce3647ca41c9000f22db7ac97271271d: Status 404 returned error can't find the container with id 2fb54e46be2b0bef792ef5c1c5664e63ce3647ca41c9000f22db7ac97271271d Nov 03 22:35:07.755931 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:07.755890 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:35:07.756129 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:07.755993 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:35:07.756129 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:07.756039 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:35:07.987436 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:07.987390 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-l96p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:35:07.987644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:07.987464 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" podUID="d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:35:08.291134 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:08.291036 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" event={"ID":"9477ee77-a2d4-4500-9bf8-5f2b8f363928","Type":"ContainerStarted","Data":"2eea585f1102ee82ddc0cf10f595754f4b3e0bb1a0266baed94d81193649add7"} Nov 03 22:35:08.291365 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:08.291209 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" event={"ID":"9477ee77-a2d4-4500-9bf8-5f2b8f363928","Type":"ContainerStarted","Data":"2fb54e46be2b0bef792ef5c1c5664e63ce3647ca41c9000f22db7ac97271271d"} Nov 03 22:35:08.410290 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:08.410230 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" podStartSLOduration=2.410215573 podStartE2EDuration="2.410215573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:08.408870491 +0000 UTC m=+1676.002902466" watchObservedRunningTime="2026-11-03 22:35:08.410215573 +0000 UTC m=+1676.004247624" Nov 03 22:35:08.816988 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:08.816946 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf"] Nov 03 22:35:08.817451 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:08.817195 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" podUID="8772e4ff-b94d-4424-b932-3a680dde9d73" containerName="istio-proxy" containerID="cri-o://f2fc48f757c2a8677e1d3fe92d9d5b3e23e7e3797696b02ae0f7304e93fff620" gracePeriod=30 Nov 03 22:35:08.996298 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:08.995779 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-l96p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 22:35:08.996298 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:08.995842 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" podUID="d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 22:35:09.001359 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.001317 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp"] Nov 03 22:35:09.012820 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.012789 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.045601 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.045561 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp"] Nov 03 22:35:09.068521 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.068413 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bc88eb47\"/\"e2e-lifecycle-model-bc88eb47-kserve-self-signed-certs\"" Nov 03 22:35:09.071010 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.070975 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-bc88eb47\"/\"default-dockercfg-slvq7\"" Nov 03 22:35:09.071240 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.071100 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bc88eb47\"/\"openshift-service-ca.crt\"" Nov 03 22:35:09.071240 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.071199 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-bc88eb47\"/\"kube-root-ca.crt\"" Nov 03 22:35:09.212663 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.212614 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-dshm\") pod \"e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.213251 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.213219 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-model-cache\") pod \"e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.213535 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.213510 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5w96\" (UniqueName: \"kubernetes.io/projected/bdf09e52-d50f-4d0e-812c-60220d5533b0-kube-api-access-r5w96\") pod \"e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.213675 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.213659 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-home\") pod \"e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.213879 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.213857 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf09e52-d50f-4d0e-812c-60220d5533b0-tls-certs\") pod \"e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.214042 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.214018 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-tmp-dir\") pod \"e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.227843 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.225463 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw" podUID="eaa113bb-18e1-4bd6-9c24-65e16319bf5a" containerName="payload-processing" containerID="cri-o://8ab977e10514a64d068fb455dc0c589bdb7999ced18a56fecaeec4920de91148" gracePeriod=25 Nov 03 22:35:09.299310 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.275563 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:09.317951 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.316742 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:09.328718 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.328603 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-model-cache\") pod \"e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.328718 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.328714 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5w96\" (UniqueName: \"kubernetes.io/projected/bdf09e52-d50f-4d0e-812c-60220d5533b0-kube-api-access-r5w96\") pod \"e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.329006 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.328745 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-home\") pod \"e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.329006 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.328789 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf09e52-d50f-4d0e-812c-60220d5533b0-tls-certs\") pod \"e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.329006 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.328816 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-tmp-dir\") pod \"e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.329006 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.328858 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-dshm\") pod \"e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.329657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.329430 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-model-cache\") pod \"e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.329657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.329530 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-tmp-dir\") pod \"e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.329895 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.329851 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-home\") pod \"e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.332993 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.332649 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-dshm\") pod \"e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.333276 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.333192 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:09.333971 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.333919 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf09e52-d50f-4d0e-812c-60220d5533b0-tls-certs\") pod \"e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.335480 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.335454 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:35:09.350459 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.350409 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f" podUID="5c54e202-d710-4a52-8c33-f773bdd018b5" containerName="payload-pre-processing" containerID="cri-o://080d1f4d0f7f4ba4bae6d6f045d772f52e476a047d8c2f0243aa6fcf90f996a8" gracePeriod=25 Nov 03 22:35:09.392111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.392038 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5w96\" (UniqueName: \"kubernetes.io/projected/bdf09e52-d50f-4d0e-812c-60220d5533b0-kube-api-access-r5w96\") pod \"e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.645280 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.644903 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:09.842270 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.842236 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw" Nov 03 22:35:09.968984 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.950768 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa113bb-18e1-4bd6-9c24-65e16319bf5a-kube-api-access-cfss7\" (UniqueName: \"kubernetes.io/projected/eaa113bb-18e1-4bd6-9c24-65e16319bf5a-kube-api-access-cfss7\") pod \"eaa113bb-18e1-4bd6-9c24-65e16319bf5a\" (UID: \"eaa113bb-18e1-4bd6-9c24-65e16319bf5a\") " Nov 03 22:35:09.968984 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.950980 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa113bb-18e1-4bd6-9c24-65e16319bf5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa113bb-18e1-4bd6-9c24-65e16319bf5a-plugins-config-volume\") pod \"eaa113bb-18e1-4bd6-9c24-65e16319bf5a\" (UID: \"eaa113bb-18e1-4bd6-9c24-65e16319bf5a\") " Nov 03 22:35:09.968984 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:09.952214 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa113bb-18e1-4bd6-9c24-65e16319bf5a-plugins-config-volume" pod "eaa113bb-18e1-4bd6-9c24-65e16319bf5a" (UID: "eaa113bb-18e1-4bd6-9c24-65e16319bf5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:10.017048 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.017004 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa113bb-18e1-4bd6-9c24-65e16319bf5a-kube-api-access-cfss7" pod "eaa113bb-18e1-4bd6-9c24-65e16319bf5a" (UID: "eaa113bb-18e1-4bd6-9c24-65e16319bf5a"). InnerVolumeSpecName "kube-api-access-cfss7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:10.054081 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.053985 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfss7\" (UniqueName: \"kubernetes.io/projected/eaa113bb-18e1-4bd6-9c24-65e16319bf5a-kube-api-access-cfss7\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:10.054081 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.054032 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaa113bb-18e1-4bd6-9c24-65e16319bf5a-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:10.107440 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.095436 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f" Nov 03 22:35:10.256669 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.256457 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c54e202-d710-4a52-8c33-f773bdd018b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c54e202-d710-4a52-8c33-f773bdd018b5-plugins-config-volume\") pod \"5c54e202-d710-4a52-8c33-f773bdd018b5\" (UID: \"5c54e202-d710-4a52-8c33-f773bdd018b5\") " Nov 03 22:35:10.256843 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.256765 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c54e202-d710-4a52-8c33-f773bdd018b5-kube-api-access-p5b4d\" (UniqueName: \"kubernetes.io/projected/5c54e202-d710-4a52-8c33-f773bdd018b5-kube-api-access-p5b4d\") pod \"5c54e202-d710-4a52-8c33-f773bdd018b5\" (UID: \"5c54e202-d710-4a52-8c33-f773bdd018b5\") " Nov 03 22:35:10.256987 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.256952 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c54e202-d710-4a52-8c33-f773bdd018b5-plugins-config-volume" pod "5c54e202-d710-4a52-8c33-f773bdd018b5" (UID: "5c54e202-d710-4a52-8c33-f773bdd018b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:10.257683 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.257608 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c54e202-d710-4a52-8c33-f773bdd018b5-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:10.260169 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.260050 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c54e202-d710-4a52-8c33-f773bdd018b5-kube-api-access-p5b4d" pod "5c54e202-d710-4a52-8c33-f773bdd018b5" (UID: "5c54e202-d710-4a52-8c33-f773bdd018b5"). InnerVolumeSpecName "kube-api-access-p5b4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:10.339830 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.339787 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp"] Nov 03 22:35:10.348255 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:35:10.348213 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdf09e52_d50f_4d0e_812c_60220d5533b0.slice/crio-c6c667ca1ed9b8c2a4325006c0ee9cb968bbd4c0340d3e84385eb4190b5960b8 WatchSource:0}: Error finding container c6c667ca1ed9b8c2a4325006c0ee9cb968bbd4c0340d3e84385eb4190b5960b8: Status 404 returned error can't find the container with id c6c667ca1ed9b8c2a4325006c0ee9cb968bbd4c0340d3e84385eb4190b5960b8 Nov 03 22:35:10.350391 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.350354 5084 generic.go:356] "Generic (PLEG): container finished" podID="eaa113bb-18e1-4bd6-9c24-65e16319bf5a" containerID="8ab977e10514a64d068fb455dc0c589bdb7999ced18a56fecaeec4920de91148" exitCode=0 Nov 03 22:35:10.350611 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.350582 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw" event={"ID":"eaa113bb-18e1-4bd6-9c24-65e16319bf5a","Type":"ContainerDied","Data":"8ab977e10514a64d068fb455dc0c589bdb7999ced18a56fecaeec4920de91148"} Nov 03 22:35:10.350736 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.350649 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw" event={"ID":"eaa113bb-18e1-4bd6-9c24-65e16319bf5a","Type":"ContainerDied","Data":"e30f2d85144e22b7d68db724735ff5a9043613d617976ce60884799ab1b8d553"} Nov 03 22:35:10.350736 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.350684 5084 scope.go:122] "RemoveContainer" containerID="8ab977e10514a64d068fb455dc0c589bdb7999ced18a56fecaeec4920de91148" Nov 03 22:35:10.352113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.352087 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw" Nov 03 22:35:10.360543 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.360512 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5b4d\" (UniqueName: \"kubernetes.io/projected/5c54e202-d710-4a52-8c33-f773bdd018b5-kube-api-access-p5b4d\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:10.375950 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.375847 5084 generic.go:356] "Generic (PLEG): container finished" podID="5c54e202-d710-4a52-8c33-f773bdd018b5" containerID="080d1f4d0f7f4ba4bae6d6f045d772f52e476a047d8c2f0243aa6fcf90f996a8" exitCode=0 Nov 03 22:35:10.381124 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.377277 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f" Nov 03 22:35:10.381766 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.381557 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f" event={"ID":"5c54e202-d710-4a52-8c33-f773bdd018b5","Type":"ContainerDied","Data":"080d1f4d0f7f4ba4bae6d6f045d772f52e476a047d8c2f0243aa6fcf90f996a8"} Nov 03 22:35:10.381766 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.381634 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f" event={"ID":"5c54e202-d710-4a52-8c33-f773bdd018b5","Type":"ContainerDied","Data":"d35707dddb597bcc50e9eefe01ef04904a041f3f92a71604645271025895cc97"} Nov 03 22:35:10.419487 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.419446 5084 scope.go:122] "RemoveContainer" containerID="8ab977e10514a64d068fb455dc0c589bdb7999ced18a56fecaeec4920de91148" Nov 03 22:35:10.419873 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:35:10.419848 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab977e10514a64d068fb455dc0c589bdb7999ced18a56fecaeec4920de91148\": container with ID starting with 8ab977e10514a64d068fb455dc0c589bdb7999ced18a56fecaeec4920de91148 not found: ID does not exist" containerID="8ab977e10514a64d068fb455dc0c589bdb7999ced18a56fecaeec4920de91148" Nov 03 22:35:10.419956 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.419884 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab977e10514a64d068fb455dc0c589bdb7999ced18a56fecaeec4920de91148"} err="failed to get container status \"8ab977e10514a64d068fb455dc0c589bdb7999ced18a56fecaeec4920de91148\": rpc error: code = NotFound desc = could not find container \"8ab977e10514a64d068fb455dc0c589bdb7999ced18a56fecaeec4920de91148\": container with ID starting with 8ab977e10514a64d068fb455dc0c589bdb7999ced18a56fecaeec4920de91148 not found: ID does not exist" Nov 03 22:35:10.419956 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.419909 5084 scope.go:122] "RemoveContainer" containerID="080d1f4d0f7f4ba4bae6d6f045d772f52e476a047d8c2f0243aa6fcf90f996a8" Nov 03 22:35:10.442681 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.442632 5084 scope.go:122] "RemoveContainer" containerID="080d1f4d0f7f4ba4bae6d6f045d772f52e476a047d8c2f0243aa6fcf90f996a8" Nov 03 22:35:10.443157 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:35:10.443133 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"080d1f4d0f7f4ba4bae6d6f045d772f52e476a047d8c2f0243aa6fcf90f996a8\": container with ID starting with 080d1f4d0f7f4ba4bae6d6f045d772f52e476a047d8c2f0243aa6fcf90f996a8 not found: ID does not exist" containerID="080d1f4d0f7f4ba4bae6d6f045d772f52e476a047d8c2f0243aa6fcf90f996a8" Nov 03 22:35:10.443220 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.443166 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"080d1f4d0f7f4ba4bae6d6f045d772f52e476a047d8c2f0243aa6fcf90f996a8"} err="failed to get container status \"080d1f4d0f7f4ba4bae6d6f045d772f52e476a047d8c2f0243aa6fcf90f996a8\": rpc error: code = NotFound desc = could not find container \"080d1f4d0f7f4ba4bae6d6f045d772f52e476a047d8c2f0243aa6fcf90f996a8\": container with ID starting with 080d1f4d0f7f4ba4bae6d6f045d772f52e476a047d8c2f0243aa6fcf90f996a8 not found: ID does not exist" Nov 03 22:35:10.690172 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.684142 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw"] Nov 03 22:35:10.690172 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.684207 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c211881d-57cbcd75b6-2lprw"] Nov 03 22:35:10.739536 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.739456 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f"] Nov 03 22:35:10.825089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:10.809220 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c211881d-5b9d45c9cf-p485f"] Nov 03 22:35:11.059295 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:11.059248 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-l96p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:35:11.059799 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:11.059336 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" podUID="d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:35:11.160103 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:11.160051 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:35:11.160715 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:11.160690 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:35:11.161903 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:11.161871 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" Nov 03 22:35:11.397013 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:11.396888 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" event={"ID":"bdf09e52-d50f-4d0e-812c-60220d5533b0","Type":"ContainerStarted","Data":"460845079ddf204883639d459655b2d9c7802e11c64bcd1d97e1624aa5309782"} Nov 03 22:35:11.397013 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:11.396947 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" event={"ID":"bdf09e52-d50f-4d0e-812c-60220d5533b0","Type":"ContainerStarted","Data":"c6c667ca1ed9b8c2a4325006c0ee9cb968bbd4c0340d3e84385eb4190b5960b8"} Nov 03 22:35:11.518519 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:11.518340 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" podStartSLOduration=3.518322033 podStartE2EDuration="3.518322033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:11.518234817 +0000 UTC m=+1679.112266827" watchObservedRunningTime="2026-11-03 22:35:11.518322033 +0000 UTC m=+1679.112354007" Nov 03 22:35:11.694149 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:11.693948 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:12.088261 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:35:12.088210 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:35:12.630729 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:12.630697 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c54e202-d710-4a52-8c33-f773bdd018b5" path="/var/lib/kubelet/pods/5c54e202-d710-4a52-8c33-f773bdd018b5/volumes" Nov 03 22:35:12.631281 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:12.631262 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa113bb-18e1-4bd6-9c24-65e16319bf5a" path="/var/lib/kubelet/pods/eaa113bb-18e1-4bd6-9c24-65e16319bf5a/volumes" Nov 03 22:35:14.131153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.131118 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:35:14.187112 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.187053 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-workload-socket\") pod \"8772e4ff-b94d-4424-b932-3a680dde9d73\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " Nov 03 22:35:14.187317 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.187201 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8772e4ff-b94d-4424-b932-3a680dde9d73-kube-api-access-ddjhz\" (UniqueName: \"kubernetes.io/projected/8772e4ff-b94d-4424-b932-3a680dde9d73-kube-api-access-ddjhz\") pod \"8772e4ff-b94d-4424-b932-3a680dde9d73\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " Nov 03 22:35:14.187317 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.187258 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-workload-certs\") pod \"8772e4ff-b94d-4424-b932-3a680dde9d73\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " Nov 03 22:35:14.187426 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.187320 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-data\") pod \"8772e4ff-b94d-4424-b932-3a680dde9d73\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " Nov 03 22:35:14.187426 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.187396 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-workload-socket" pod "8772e4ff-b94d-4424-b932-3a680dde9d73" (UID: "8772e4ff-b94d-4424-b932-3a680dde9d73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:14.187576 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.187448 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-podinfo\") pod \"8772e4ff-b94d-4424-b932-3a680dde9d73\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " Nov 03 22:35:14.187641 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.187585 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-data" pod "8772e4ff-b94d-4424-b932-3a680dde9d73" (UID: "8772e4ff-b94d-4424-b932-3a680dde9d73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:14.187641 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.187595 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-workload-certs" pod "8772e4ff-b94d-4424-b932-3a680dde9d73" (UID: "8772e4ff-b94d-4424-b932-3a680dde9d73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:14.187865 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.187840 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8772e4ff-b94d-4424-b932-3a680dde9d73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8772e4ff-b94d-4424-b932-3a680dde9d73-istiod-ca-cert\") pod \"8772e4ff-b94d-4424-b932-3a680dde9d73\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " Nov 03 22:35:14.188673 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.188054 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-envoy\") pod \"8772e4ff-b94d-4424-b932-3a680dde9d73\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " Nov 03 22:35:14.188673 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.188116 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-token\" (UniqueName: \"kubernetes.io/projected/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-token\") pod \"8772e4ff-b94d-4424-b932-3a680dde9d73\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " Nov 03 22:35:14.188673 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.188632 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8772e4ff-b94d-4424-b932-3a680dde9d73-istiod-ca-cert" pod "8772e4ff-b94d-4424-b932-3a680dde9d73" (UID: "8772e4ff-b94d-4424-b932-3a680dde9d73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:14.189043 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.188886 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-credential-socket" pod "8772e4ff-b94d-4424-b932-3a680dde9d73" (UID: "8772e4ff-b94d-4424-b932-3a680dde9d73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:14.189043 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.188355 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-credential-socket\") pod \"8772e4ff-b94d-4424-b932-3a680dde9d73\" (UID: \"8772e4ff-b94d-4424-b932-3a680dde9d73\") " Nov 03 22:35:14.189896 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.189873 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.190225 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.189902 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.190225 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.189916 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.190225 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.189925 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.190225 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.189933 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8772e4ff-b94d-4424-b932-3a680dde9d73-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.190888 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.190863 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8772e4ff-b94d-4424-b932-3a680dde9d73-kube-api-access-ddjhz" pod "8772e4ff-b94d-4424-b932-3a680dde9d73" (UID: "8772e4ff-b94d-4424-b932-3a680dde9d73"). InnerVolumeSpecName "kube-api-access-ddjhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:14.190980 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.190944 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-podinfo" pod "8772e4ff-b94d-4424-b932-3a680dde9d73" (UID: "8772e4ff-b94d-4424-b932-3a680dde9d73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:14.191240 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.191210 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-envoy" pod "8772e4ff-b94d-4424-b932-3a680dde9d73" (UID: "8772e4ff-b94d-4424-b932-3a680dde9d73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:14.191378 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.191361 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-token" pod "8772e4ff-b94d-4424-b932-3a680dde9d73" (UID: "8772e4ff-b94d-4424-b932-3a680dde9d73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:14.291644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.291605 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddjhz\" (UniqueName: \"kubernetes.io/projected/8772e4ff-b94d-4424-b932-3a680dde9d73-kube-api-access-ddjhz\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.291644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.291643 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.291921 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.291657 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.291921 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.291669 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8772e4ff-b94d-4424-b932-3a680dde9d73-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:14.436786 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.436689 5084 generic.go:356] "Generic (PLEG): container finished" podID="8772e4ff-b94d-4424-b932-3a680dde9d73" containerID="f2fc48f757c2a8677e1d3fe92d9d5b3e23e7e3797696b02ae0f7304e93fff620" exitCode=0 Nov 03 22:35:14.437164 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.437136 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" event={"ID":"8772e4ff-b94d-4424-b932-3a680dde9d73","Type":"ContainerDied","Data":"f2fc48f757c2a8677e1d3fe92d9d5b3e23e7e3797696b02ae0f7304e93fff620"} Nov 03 22:35:14.437402 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.437379 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" event={"ID":"8772e4ff-b94d-4424-b932-3a680dde9d73","Type":"ContainerDied","Data":"c7c1583640ba15281965b7715a546eb85d5bb1ac6b22f39675010cdd6d488b74"} Nov 03 22:35:14.437535 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.437521 5084 scope.go:122] "RemoveContainer" containerID="f2fc48f757c2a8677e1d3fe92d9d5b3e23e7e3797696b02ae0f7304e93fff620" Nov 03 22:35:14.437734 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.437709 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf" Nov 03 22:35:14.452252 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.452226 5084 scope.go:122] "RemoveContainer" containerID="f2fc48f757c2a8677e1d3fe92d9d5b3e23e7e3797696b02ae0f7304e93fff620" Nov 03 22:35:14.452667 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:35:14.452632 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2fc48f757c2a8677e1d3fe92d9d5b3e23e7e3797696b02ae0f7304e93fff620\": container with ID starting with f2fc48f757c2a8677e1d3fe92d9d5b3e23e7e3797696b02ae0f7304e93fff620 not found: ID does not exist" containerID="f2fc48f757c2a8677e1d3fe92d9d5b3e23e7e3797696b02ae0f7304e93fff620" Nov 03 22:35:14.452712 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.452680 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2fc48f757c2a8677e1d3fe92d9d5b3e23e7e3797696b02ae0f7304e93fff620"} err="failed to get container status \"f2fc48f757c2a8677e1d3fe92d9d5b3e23e7e3797696b02ae0f7304e93fff620\": rpc error: code = NotFound desc = could not find container \"f2fc48f757c2a8677e1d3fe92d9d5b3e23e7e3797696b02ae0f7304e93fff620\": container with ID starting with f2fc48f757c2a8677e1d3fe92d9d5b3e23e7e3797696b02ae0f7304e93fff620 not found: ID does not exist" Nov 03 22:35:14.608021 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.607985 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf"] Nov 03 22:35:14.612169 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.612130 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-2rhdf"] Nov 03 22:35:14.629910 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.629872 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8772e4ff-b94d-4424-b932-3a680dde9d73" path="/var/lib/kubelet/pods/8772e4ff-b94d-4424-b932-3a680dde9d73/volumes" Nov 03 22:35:14.767887 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.767840 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz"] Nov 03 22:35:14.768575 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.768555 5084 cpu_manager.go:414] "containerMap: removing container" podUID="8772e4ff-b94d-4424-b932-3a680dde9d73" containerName="istio-proxy" Nov 03 22:35:14.768686 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.768575 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8772e4ff-b94d-4424-b932-3a680dde9d73" containerName="istio-proxy" Nov 03 22:35:14.768686 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.768589 5084 cpu_manager.go:414] "containerMap: removing container" podUID="eaa113bb-18e1-4bd6-9c24-65e16319bf5a" containerName="payload-processing" Nov 03 22:35:14.768686 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.768594 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa113bb-18e1-4bd6-9c24-65e16319bf5a" containerName="payload-processing" Nov 03 22:35:14.768686 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.768610 5084 cpu_manager.go:414] "containerMap: removing container" podUID="5c54e202-d710-4a52-8c33-f773bdd018b5" containerName="payload-pre-processing" Nov 03 22:35:14.768686 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.768615 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c54e202-d710-4a52-8c33-f773bdd018b5" containerName="payload-pre-processing" Nov 03 22:35:14.768932 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.768743 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="8772e4ff-b94d-4424-b932-3a680dde9d73" containerName="istio-proxy" Nov 03 22:35:14.768932 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.768750 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa113bb-18e1-4bd6-9c24-65e16319bf5a" containerName="payload-processing" Nov 03 22:35:14.768932 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.768758 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c54e202-d710-4a52-8c33-f773bdd018b5" containerName="payload-pre-processing" Nov 03 22:35:14.774457 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.774425 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.799260 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.799217 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.799463 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.799353 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.799463 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.799398 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.799463 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.799434 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.799737 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.799595 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44296\" (UniqueName: \"kubernetes.io/projected/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-kube-api-access-44296\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.799737 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.799641 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.799848 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.799739 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.799848 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.799782 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.799848 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.799807 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.828097 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.827694 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz"] Nov 03 22:35:14.901890 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.901840 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.902191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.901909 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.902191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.901952 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.902191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.902181 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.902378 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.902261 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.902378 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.902306 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.902378 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.902360 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.902544 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.902428 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44296\" (UniqueName: \"kubernetes.io/projected/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-kube-api-access-44296\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.902544 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.902460 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.902544 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.902460 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-istio-data\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.902692 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.902650 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-workload-certs\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.902838 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.902815 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-credential-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.902931 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.902911 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-workload-socket\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.903875 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.903602 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.904899 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.904866 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.905644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.905620 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.962604 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.962554 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44296\" (UniqueName: \"kubernetes.io/projected/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-kube-api-access-44296\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:14.963423 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:14.963386 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cebf3c1-c5fb-4feb-beac-7cf2e8a01134-istio-token\") pod \"maas-default-gateway-openshift-default-7d64fcddb7-kgjwz\" (UID: \"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:15.096044 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:15.095949 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:15.545889 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:15.545854 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz"] Nov 03 22:35:15.546547 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:35:15.546520 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cebf3c1_c5fb_4feb_beac_7cf2e8a01134.slice/crio-6cf6966d248ec0955955e5259d97a7e207033413f7b9517adfdb97dd5e075d40 WatchSource:0}: Error finding container 6cf6966d248ec0955955e5259d97a7e207033413f7b9517adfdb97dd5e075d40: Status 404 returned error can't find the container with id 6cf6966d248ec0955955e5259d97a7e207033413f7b9517adfdb97dd5e075d40 Nov 03 22:35:15.549347 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:15.549308 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:35:15.549445 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:15.549382 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:35:15.549445 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:15.549412 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:35:15.853043 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:15.852620 5084 scope.go:122] "RemoveContainer" containerID="defd66f90b7291da843bc2c77f1e1db6267bef309cefeb4fe89817179c52ec7b" Nov 03 22:35:15.865617 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:15.865589 5084 scope.go:122] "RemoveContainer" containerID="a8c8ebce40a0d5b97422231f26416be8760ac66c96512e3e9c8256e211ee963d" Nov 03 22:35:15.877521 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:15.877496 5084 scope.go:122] "RemoveContainer" containerID="1cf8491e0ded044ba5a6264da9ac159ca53f0c52b888468b611b1d2bbae3d30e" Nov 03 22:35:15.891543 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:15.891515 5084 scope.go:122] "RemoveContainer" containerID="10a731127f79eb2a29d49ada679c91833d0671e8ac868b2415dd4502a394b443" Nov 03 22:35:16.488766 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:16.488730 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" event={"ID":"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134","Type":"ContainerStarted","Data":"c57a9505ca5abba7cba69ec23146cdc34dd212594af61e034111ded837ad3b75"} Nov 03 22:35:16.488998 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:16.488787 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" event={"ID":"8cebf3c1-c5fb-4feb-beac-7cf2e8a01134","Type":"ContainerStarted","Data":"6cf6966d248ec0955955e5259d97a7e207033413f7b9517adfdb97dd5e075d40"} Nov 03 22:35:16.789326 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:16.730005 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" podStartSLOduration=2.7299816850000003 podStartE2EDuration="2.729981685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:16.624939574 +0000 UTC m=+1684.218971549" watchObservedRunningTime="2026-11-03 22:35:16.729981685 +0000 UTC m=+1684.324013659" Nov 03 22:35:16.952133 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:16.947948 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv"] Nov 03 22:35:16.963238 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:16.958177 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv" Nov 03 22:35:17.035527 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:17.025418 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-37b70b\"" Nov 03 22:35:17.035527 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:17.025644 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-37b70b-dockercfg-tfx4z\"" Nov 03 22:35:17.120165 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:17.107942 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:17.120949 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:17.120913 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:35:17.134090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:17.121164 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" podUID="8cebf3c1-c5fb-4feb-beac-7cf2e8a01134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 22:35:17.170421 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:17.160308 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9qd5\" (UniqueName: \"kubernetes.io/projected/7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7-kube-api-access-b9qd5\") pod \"payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv\" (UID: \"7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv" Nov 03 22:35:17.170421 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:17.160485 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv\" (UID: \"7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv" Nov 03 22:35:17.281088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:17.279941 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9qd5\" (UniqueName: \"kubernetes.io/projected/7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7-kube-api-access-b9qd5\") pod \"payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv\" (UID: \"7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv" Nov 03 22:35:17.281088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:17.280080 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv\" (UID: \"7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv" Nov 03 22:35:17.282138 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:17.281653 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv\" (UID: \"7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv" Nov 03 22:35:17.366089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:17.358663 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv"] Nov 03 22:35:17.572091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:17.565932 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9qd5\" (UniqueName: \"kubernetes.io/projected/7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7-kube-api-access-b9qd5\") pod \"payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv\" (UID: \"7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv" Nov 03 22:35:17.577348 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:17.577308 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv" Nov 03 22:35:17.920107 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:17.915378 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922"] Nov 03 22:35:17.932366 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:17.932326 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922"] Nov 03 22:35:17.932533 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:17.932473 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922" Nov 03 22:35:18.032916 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:18.032872 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922\" (UID: \"1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922" Nov 03 22:35:18.033147 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:18.033022 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mrqw\" (UniqueName: \"kubernetes.io/projected/1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5-kube-api-access-7mrqw\") pod \"payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922\" (UID: \"1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922" Nov 03 22:35:18.098105 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:18.096788 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:35:18.098105 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:18.096850 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" podUID="8cebf3c1-c5fb-4feb-beac-7cf2e8a01134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 22:35:18.134498 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:18.134405 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mrqw\" (UniqueName: \"kubernetes.io/projected/1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5-kube-api-access-7mrqw\") pod \"payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922\" (UID: \"1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922" Nov 03 22:35:18.134675 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:18.134612 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922\" (UID: \"1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922" Nov 03 22:35:18.135404 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:18.135379 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922\" (UID: \"1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922" Nov 03 22:35:18.192693 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:18.192659 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mrqw\" (UniqueName: \"kubernetes.io/projected/1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5-kube-api-access-7mrqw\") pod \"payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922\" (UID: \"1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922" Nov 03 22:35:18.334457 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:18.334417 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922" Nov 03 22:35:18.347761 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:18.347681 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv"] Nov 03 22:35:18.352687 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:35:18.352380 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aea9fcd_0d68_4fbb_9f7f_05972a33dfc7.slice/crio-2a394dcabb6963210c4dd3a23ad2c92a7ba99b2362c332d8cb84c6e5f27f97bd WatchSource:0}: Error finding container 2a394dcabb6963210c4dd3a23ad2c92a7ba99b2362c332d8cb84c6e5f27f97bd: Status 404 returned error can't find the container with id 2a394dcabb6963210c4dd3a23ad2c92a7ba99b2362c332d8cb84c6e5f27f97bd Nov 03 22:35:18.787229 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:18.787197 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922"] Nov 03 22:35:18.894678 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:18.894637 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv" event={"ID":"7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7","Type":"ContainerStarted","Data":"2a76f83baf1a5d8543af6c2cfab5190d8e217c7cf46b75efc5cde5b50435916e"} Nov 03 22:35:18.895332 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:18.895299 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv" event={"ID":"7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7","Type":"ContainerStarted","Data":"2a394dcabb6963210c4dd3a23ad2c92a7ba99b2362c332d8cb84c6e5f27f97bd"} Nov 03 22:35:18.895479 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:18.895351 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv" Nov 03 22:35:18.896720 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:18.896695 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922" event={"ID":"1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5","Type":"ContainerStarted","Data":"e603fe3105ffdf25935c23d9541419b95262a28daeb6a634512cbe9e3e755a53"} Nov 03 22:35:19.013710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:19.013635 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv" podStartSLOduration=3.013616431 podStartE2EDuration="3.013616431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:19.01182702 +0000 UTC m=+1686.605858994" watchObservedRunningTime="2026-11-03 22:35:19.013616431 +0000 UTC m=+1686.607648404" Nov 03 22:35:19.096652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:19.096558 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:35:19.096652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:19.096623 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" podUID="8cebf3c1-c5fb-4feb-beac-7cf2e8a01134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 22:35:19.690883 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:19.690840 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:19.704244 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:19.704210 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:19.907231 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:19.907179 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922" event={"ID":"1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5","Type":"ContainerStarted","Data":"f858dfd92eb55ef8d24080f33062975b4450820dacc2db1a23870be12cfafc7a"} Nov 03 22:35:19.907410 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:19.907253 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922" Nov 03 22:35:19.919303 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:19.919271 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:35:20.031648 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:20.031574 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922" podStartSLOduration=3.031555534 podStartE2EDuration="3.031555534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:20.026334499 +0000 UTC m=+1687.620366473" watchObservedRunningTime="2026-11-03 22:35:20.031555534 +0000 UTC m=+1687.625587603" Nov 03 22:35:20.096415 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:20.096370 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:35:20.096577 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:20.096445 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" podUID="8cebf3c1-c5fb-4feb-beac-7cf2e8a01134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 22:35:21.096733 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:21.096690 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:35:21.097170 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:21.096752 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" podUID="8cebf3c1-c5fb-4feb-beac-7cf2e8a01134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 22:35:22.097216 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:22.097176 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:35:22.097818 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:22.097242 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" podUID="8cebf3c1-c5fb-4feb-beac-7cf2e8a01134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 22:35:23.096891 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:23.096851 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:35:23.097097 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:23.096913 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" podUID="8cebf3c1-c5fb-4feb-beac-7cf2e8a01134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 22:35:24.097217 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:24.097176 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Nov 03 22:35:24.097626 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:24.097248 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" podUID="8cebf3c1-c5fb-4feb-beac-7cf2e8a01134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Nov 03 22:35:25.267135 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:25.267040 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:25.267678 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:25.267654 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:25.268603 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:25.268581 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" Nov 03 22:35:25.901786 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:25.901732 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc"] Nov 03 22:35:25.902018 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:25.901993 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerName="istio-proxy" containerID="cri-o://98c682ae79b807db44a82e334201f2de3acdace88d19427e0426ef41f98f0482" gracePeriod=30 Nov 03 22:35:30.919622 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:30.919591 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922" Nov 03 22:35:30.920104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:30.919676 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv" Nov 03 22:35:31.032195 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.032159 5084 generic.go:356] "Generic (PLEG): container finished" podID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerID="98c682ae79b807db44a82e334201f2de3acdace88d19427e0426ef41f98f0482" exitCode=0 Nov 03 22:35:31.032382 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.032229 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" event={"ID":"5350dd65-ca9c-4a6f-8356-9f925f84f864","Type":"ContainerDied","Data":"98c682ae79b807db44a82e334201f2de3acdace88d19427e0426ef41f98f0482"} Nov 03 22:35:31.285508 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.285472 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:35:31.457469 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.457424 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-workload-certs\") pod \"5350dd65-ca9c-4a6f-8356-9f925f84f864\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " Nov 03 22:35:31.457693 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.457538 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-podinfo\") pod \"5350dd65-ca9c-4a6f-8356-9f925f84f864\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " Nov 03 22:35:31.457693 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.457599 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-data\") pod \"5350dd65-ca9c-4a6f-8356-9f925f84f864\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " Nov 03 22:35:31.457693 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.457651 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5350dd65-ca9c-4a6f-8356-9f925f84f864-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5350dd65-ca9c-4a6f-8356-9f925f84f864-istiod-ca-cert\") pod \"5350dd65-ca9c-4a6f-8356-9f925f84f864\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " Nov 03 22:35:31.457875 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.457784 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-envoy\") pod \"5350dd65-ca9c-4a6f-8356-9f925f84f864\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " Nov 03 22:35:31.457875 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.457822 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-workload-socket\") pod \"5350dd65-ca9c-4a6f-8356-9f925f84f864\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " Nov 03 22:35:31.457875 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.457851 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-token\" (UniqueName: \"kubernetes.io/projected/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-token\") pod \"5350dd65-ca9c-4a6f-8356-9f925f84f864\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " Nov 03 22:35:31.458027 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.457907 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-credential-socket\") pod \"5350dd65-ca9c-4a6f-8356-9f925f84f864\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " Nov 03 22:35:31.458027 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.457933 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5350dd65-ca9c-4a6f-8356-9f925f84f864-kube-api-access-42sdj\" (UniqueName: \"kubernetes.io/projected/5350dd65-ca9c-4a6f-8356-9f925f84f864-kube-api-access-42sdj\") pod \"5350dd65-ca9c-4a6f-8356-9f925f84f864\" (UID: \"5350dd65-ca9c-4a6f-8356-9f925f84f864\") " Nov 03 22:35:31.458510 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.458474 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-workload-socket" pod "5350dd65-ca9c-4a6f-8356-9f925f84f864" (UID: "5350dd65-ca9c-4a6f-8356-9f925f84f864"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:31.458980 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.458687 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-credential-socket" pod "5350dd65-ca9c-4a6f-8356-9f925f84f864" (UID: "5350dd65-ca9c-4a6f-8356-9f925f84f864"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:31.458980 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.458719 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-data" pod "5350dd65-ca9c-4a6f-8356-9f925f84f864" (UID: "5350dd65-ca9c-4a6f-8356-9f925f84f864"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:31.458980 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.458741 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-workload-certs" pod "5350dd65-ca9c-4a6f-8356-9f925f84f864" (UID: "5350dd65-ca9c-4a6f-8356-9f925f84f864"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:31.458980 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.458815 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5350dd65-ca9c-4a6f-8356-9f925f84f864-istiod-ca-cert" pod "5350dd65-ca9c-4a6f-8356-9f925f84f864" (UID: "5350dd65-ca9c-4a6f-8356-9f925f84f864"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:31.459517 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.459487 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:31.459852 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.459522 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:31.459852 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.459538 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:31.459852 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.459552 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:31.459852 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.459565 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5350dd65-ca9c-4a6f-8356-9f925f84f864-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:31.461181 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.461154 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-envoy" pod "5350dd65-ca9c-4a6f-8356-9f925f84f864" (UID: "5350dd65-ca9c-4a6f-8356-9f925f84f864"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:31.461282 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.461222 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-token" pod "5350dd65-ca9c-4a6f-8356-9f925f84f864" (UID: "5350dd65-ca9c-4a6f-8356-9f925f84f864"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:31.461459 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.461437 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-podinfo" pod "5350dd65-ca9c-4a6f-8356-9f925f84f864" (UID: "5350dd65-ca9c-4a6f-8356-9f925f84f864"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:31.462885 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.462858 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5350dd65-ca9c-4a6f-8356-9f925f84f864-kube-api-access-42sdj" pod "5350dd65-ca9c-4a6f-8356-9f925f84f864" (UID: "5350dd65-ca9c-4a6f-8356-9f925f84f864"). InnerVolumeSpecName "kube-api-access-42sdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:31.561041 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.560995 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:31.561041 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.561034 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:31.561041 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.561044 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42sdj\" (UniqueName: \"kubernetes.io/projected/5350dd65-ca9c-4a6f-8356-9f925f84f864-kube-api-access-42sdj\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:31.561382 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:31.561076 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5350dd65-ca9c-4a6f-8356-9f925f84f864-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:32.042605 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:32.042561 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" event={"ID":"5350dd65-ca9c-4a6f-8356-9f925f84f864","Type":"ContainerDied","Data":"5845c4aab092fdd878aea6669c8eb5dc33af36a15f6061ba825e8b66ca9aced4"} Nov 03 22:35:32.042605 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:32.042585 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc" Nov 03 22:35:32.043197 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:32.042632 5084 scope.go:122] "RemoveContainer" containerID="98c682ae79b807db44a82e334201f2de3acdace88d19427e0426ef41f98f0482" Nov 03 22:35:32.213435 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:32.213396 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc"] Nov 03 22:35:32.264459 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:32.264412 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-nldrc"] Nov 03 22:35:32.632906 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:32.632860 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" path="/var/lib/kubelet/pods/5350dd65-ca9c-4a6f-8356-9f925f84f864/volumes" Nov 03 22:35:33.374839 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:33.374796 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh"] Nov 03 22:35:33.375449 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:33.375015 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" podUID="59460ed3-3f95-4092-abe1-a42780972f29" containerName="maas-api" containerID="cri-o://cb3ecf5cbf111abfd5ae9a5f62c762065919f7b47a3a16bd7afef18f59b92e21" gracePeriod=30 Nov 03 22:35:33.879697 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:33.879663 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:35:33.887697 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:33.887652 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59460ed3-3f95-4092-abe1-a42780972f29-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59460ed3-3f95-4092-abe1-a42780972f29-openshift-service-ca\") pod \"59460ed3-3f95-4092-abe1-a42780972f29\" (UID: \"59460ed3-3f95-4092-abe1-a42780972f29\") " Nov 03 22:35:33.887891 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:33.887809 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59460ed3-3f95-4092-abe1-a42780972f29-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59460ed3-3f95-4092-abe1-a42780972f29-maas-api-tls\") pod \"59460ed3-3f95-4092-abe1-a42780972f29\" (UID: \"59460ed3-3f95-4092-abe1-a42780972f29\") " Nov 03 22:35:33.887891 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:33.887871 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59460ed3-3f95-4092-abe1-a42780972f29-kube-api-access-tjz8p\" (UniqueName: \"kubernetes.io/projected/59460ed3-3f95-4092-abe1-a42780972f29-kube-api-access-tjz8p\") pod \"59460ed3-3f95-4092-abe1-a42780972f29\" (UID: \"59460ed3-3f95-4092-abe1-a42780972f29\") " Nov 03 22:35:33.888035 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:33.888016 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59460ed3-3f95-4092-abe1-a42780972f29-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59460ed3-3f95-4092-abe1-a42780972f29-metrics-certs\") pod \"59460ed3-3f95-4092-abe1-a42780972f29\" (UID: \"59460ed3-3f95-4092-abe1-a42780972f29\") " Nov 03 22:35:33.888207 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:33.888182 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59460ed3-3f95-4092-abe1-a42780972f29-openshift-service-ca" pod "59460ed3-3f95-4092-abe1-a42780972f29" (UID: "59460ed3-3f95-4092-abe1-a42780972f29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:33.889284 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:33.889241 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59460ed3-3f95-4092-abe1-a42780972f29-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:33.891028 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:33.890992 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59460ed3-3f95-4092-abe1-a42780972f29-maas-api-tls" pod "59460ed3-3f95-4092-abe1-a42780972f29" (UID: "59460ed3-3f95-4092-abe1-a42780972f29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:35:33.891217 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:33.891038 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59460ed3-3f95-4092-abe1-a42780972f29-kube-api-access-tjz8p" pod "59460ed3-3f95-4092-abe1-a42780972f29" (UID: "59460ed3-3f95-4092-abe1-a42780972f29"). InnerVolumeSpecName "kube-api-access-tjz8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:33.892612 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:33.892540 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59460ed3-3f95-4092-abe1-a42780972f29-metrics-certs" pod "59460ed3-3f95-4092-abe1-a42780972f29" (UID: "59460ed3-3f95-4092-abe1-a42780972f29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:35:34.004303 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:33.992408 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59460ed3-3f95-4092-abe1-a42780972f29-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:34.004303 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:33.992453 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59460ed3-3f95-4092-abe1-a42780972f29-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:34.004303 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:33.992469 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjz8p\" (UniqueName: \"kubernetes.io/projected/59460ed3-3f95-4092-abe1-a42780972f29-kube-api-access-tjz8p\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:34.133886 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:34.133785 5084 generic.go:356] "Generic (PLEG): container finished" podID="59460ed3-3f95-4092-abe1-a42780972f29" containerID="cb3ecf5cbf111abfd5ae9a5f62c762065919f7b47a3a16bd7afef18f59b92e21" exitCode=0 Nov 03 22:35:34.134151 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:34.133950 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" event={"ID":"59460ed3-3f95-4092-abe1-a42780972f29","Type":"ContainerDied","Data":"cb3ecf5cbf111abfd5ae9a5f62c762065919f7b47a3a16bd7afef18f59b92e21"} Nov 03 22:35:34.134151 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:34.134006 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" event={"ID":"59460ed3-3f95-4092-abe1-a42780972f29","Type":"ContainerDied","Data":"e504930597bf831b83a7dea0c8b92fb3220869b32a1bd48ff54293c14cdd4cea"} Nov 03 22:35:34.134151 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:34.134041 5084 scope.go:122] "RemoveContainer" containerID="cb3ecf5cbf111abfd5ae9a5f62c762065919f7b47a3a16bd7afef18f59b92e21" Nov 03 22:35:34.134363 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:34.134201 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh" Nov 03 22:35:34.166798 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:34.166771 5084 scope.go:122] "RemoveContainer" containerID="cb3ecf5cbf111abfd5ae9a5f62c762065919f7b47a3a16bd7afef18f59b92e21" Nov 03 22:35:34.167252 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:35:34.167224 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb3ecf5cbf111abfd5ae9a5f62c762065919f7b47a3a16bd7afef18f59b92e21\": container with ID starting with cb3ecf5cbf111abfd5ae9a5f62c762065919f7b47a3a16bd7afef18f59b92e21 not found: ID does not exist" containerID="cb3ecf5cbf111abfd5ae9a5f62c762065919f7b47a3a16bd7afef18f59b92e21" Nov 03 22:35:34.167361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:34.167263 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb3ecf5cbf111abfd5ae9a5f62c762065919f7b47a3a16bd7afef18f59b92e21"} err="failed to get container status \"cb3ecf5cbf111abfd5ae9a5f62c762065919f7b47a3a16bd7afef18f59b92e21\": rpc error: code = NotFound desc = could not find container \"cb3ecf5cbf111abfd5ae9a5f62c762065919f7b47a3a16bd7afef18f59b92e21\": container with ID starting with cb3ecf5cbf111abfd5ae9a5f62c762065919f7b47a3a16bd7afef18f59b92e21 not found: ID does not exist" Nov 03 22:35:34.417898 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:34.417792 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh"] Nov 03 22:35:34.425029 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:34.424982 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c211881d-cd7c5c787-lhzdh"] Nov 03 22:35:34.677104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:34.676990 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59460ed3-3f95-4092-abe1-a42780972f29" path="/var/lib/kubelet/pods/59460ed3-3f95-4092-abe1-a42780972f29/volumes" Nov 03 22:35:36.048799 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:36.048756 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs"] Nov 03 22:35:36.049354 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:36.048994 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" podUID="18e734ea-1f54-4603-9fd9-3261ba6d4ec7" containerName="istio-proxy" containerID="cri-o://751113cfbd3567b732578d94df98a04dd3c74c25941308cf9d2bc41152e1f18c" gracePeriod=30 Nov 03 22:35:39.574735 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:39.574687 5084 patch_prober.go:29] interesting pod/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:35:39.575191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:39.574764 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" podUID="18e734ea-1f54-4603-9fd9-3261ba6d4ec7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:35:41.699115 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.696367 5084 generic.go:356] "Generic (PLEG): container finished" podID="18e734ea-1f54-4603-9fd9-3261ba6d4ec7" containerID="751113cfbd3567b732578d94df98a04dd3c74c25941308cf9d2bc41152e1f18c" exitCode=0 Nov 03 22:35:41.699115 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.696505 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" event={"ID":"18e734ea-1f54-4603-9fd9-3261ba6d4ec7","Type":"ContainerDied","Data":"751113cfbd3567b732578d94df98a04dd3c74c25941308cf9d2bc41152e1f18c"} Nov 03 22:35:41.699115 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.696621 5084 scope.go:122] "RemoveContainer" containerID="a5d01c285a29b9b9289e19676165eafd86047db93daacdc38be306402a49aaba" Nov 03 22:35:41.739707 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.739679 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:35:41.763769 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.763729 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-podinfo\") pod \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " Nov 03 22:35:41.763970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.763797 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-envoy\") pod \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " Nov 03 22:35:41.763970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.763885 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-workload-certs\") pod \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " Nov 03 22:35:41.763970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.763929 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istiod-ca-cert\") pod \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " Nov 03 22:35:41.763970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.763967 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-data\") pod \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " Nov 03 22:35:41.764220 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.764015 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-token\" (UniqueName: \"kubernetes.io/projected/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-token\") pod \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " Nov 03 22:35:41.764220 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.764142 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-workload-socket\") pod \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " Nov 03 22:35:41.764388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.764262 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-workload-certs" pod "18e734ea-1f54-4603-9fd9-3261ba6d4ec7" (UID: "18e734ea-1f54-4603-9fd9-3261ba6d4ec7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:41.764388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.764277 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-data" pod "18e734ea-1f54-4603-9fd9-3261ba6d4ec7" (UID: "18e734ea-1f54-4603-9fd9-3261ba6d4ec7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:41.764388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.764316 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-kube-api-access-6kslf\" (UniqueName: \"kubernetes.io/projected/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-kube-api-access-6kslf\") pod \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " Nov 03 22:35:41.764539 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.764396 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-credential-socket\") pod \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\" (UID: \"18e734ea-1f54-4603-9fd9-3261ba6d4ec7\") " Nov 03 22:35:41.764539 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.764436 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-workload-socket" pod "18e734ea-1f54-4603-9fd9-3261ba6d4ec7" (UID: "18e734ea-1f54-4603-9fd9-3261ba6d4ec7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:41.764539 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.764443 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istiod-ca-cert" pod "18e734ea-1f54-4603-9fd9-3261ba6d4ec7" (UID: "18e734ea-1f54-4603-9fd9-3261ba6d4ec7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:41.764746 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.764697 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-credential-socket" pod "18e734ea-1f54-4603-9fd9-3261ba6d4ec7" (UID: "18e734ea-1f54-4603-9fd9-3261ba6d4ec7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:41.765281 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.765234 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:41.765281 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.765263 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:41.765555 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.765290 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:41.765555 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.765352 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:41.765555 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.765367 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:41.766531 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.766501 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-podinfo" pod "18e734ea-1f54-4603-9fd9-3261ba6d4ec7" (UID: "18e734ea-1f54-4603-9fd9-3261ba6d4ec7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:41.766715 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.766692 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-token" pod "18e734ea-1f54-4603-9fd9-3261ba6d4ec7" (UID: "18e734ea-1f54-4603-9fd9-3261ba6d4ec7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:41.766783 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.766749 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-envoy" pod "18e734ea-1f54-4603-9fd9-3261ba6d4ec7" (UID: "18e734ea-1f54-4603-9fd9-3261ba6d4ec7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:41.766871 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.766842 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-kube-api-access-6kslf" pod "18e734ea-1f54-4603-9fd9-3261ba6d4ec7" (UID: "18e734ea-1f54-4603-9fd9-3261ba6d4ec7"). InnerVolumeSpecName "kube-api-access-6kslf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:41.855184 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.855130 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh"] Nov 03 22:35:41.856268 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.856239 5084 cpu_manager.go:414] "containerMap: removing container" podUID="59460ed3-3f95-4092-abe1-a42780972f29" containerName="maas-api" Nov 03 22:35:41.856364 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.856270 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59460ed3-3f95-4092-abe1-a42780972f29" containerName="maas-api" Nov 03 22:35:41.856364 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.856321 5084 cpu_manager.go:414] "containerMap: removing container" podUID="18e734ea-1f54-4603-9fd9-3261ba6d4ec7" containerName="istio-proxy" Nov 03 22:35:41.856364 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.856329 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e734ea-1f54-4603-9fd9-3261ba6d4ec7" containerName="istio-proxy" Nov 03 22:35:41.856364 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.856353 5084 cpu_manager.go:414] "containerMap: removing container" podUID="18e734ea-1f54-4603-9fd9-3261ba6d4ec7" containerName="istio-proxy" Nov 03 22:35:41.856364 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.856360 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e734ea-1f54-4603-9fd9-3261ba6d4ec7" containerName="istio-proxy" Nov 03 22:35:41.856652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.856384 5084 cpu_manager.go:414] "containerMap: removing container" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerName="istio-proxy" Nov 03 22:35:41.856652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.856391 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerName="istio-proxy" Nov 03 22:35:41.856652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.856556 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="59460ed3-3f95-4092-abe1-a42780972f29" containerName="maas-api" Nov 03 22:35:41.856652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.856578 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e734ea-1f54-4603-9fd9-3261ba6d4ec7" containerName="istio-proxy" Nov 03 22:35:41.856652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.856589 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e734ea-1f54-4603-9fd9-3261ba6d4ec7" containerName="istio-proxy" Nov 03 22:35:41.856652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.856602 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="5350dd65-ca9c-4a6f-8356-9f925f84f864" containerName="istio-proxy" Nov 03 22:35:41.862587 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.862142 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:41.869456 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.869410 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:41.869456 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.869457 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:41.869697 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.869473 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kslf\" (UniqueName: \"kubernetes.io/projected/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-kube-api-access-6kslf\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:41.869697 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.869487 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e734ea-1f54-4603-9fd9-3261ba6d4ec7-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:35:41.915496 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.915453 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh"] Nov 03 22:35:41.947576 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.947542 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ca6853-openshift-default-dockercfg-7mbw8\"" Nov 03 22:35:41.982160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.980128 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd2bl\" (UniqueName: \"kubernetes.io/projected/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-kube-api-access-nd2bl\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:41.982160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.980222 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-workload-socket\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:41.982160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.980253 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-credential-socket\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:41.982160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.980284 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-data\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:41.982160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.980311 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-workload-certs\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:41.982160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.980349 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-token\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:41.982160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.980428 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-envoy\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:41.982160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.980554 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istiod-ca-cert\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:41.982160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:41.980675 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-podinfo\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.104759 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.103560 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd2bl\" (UniqueName: \"kubernetes.io/projected/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-kube-api-access-nd2bl\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.104759 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.103655 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-workload-socket\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.104759 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.103684 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-credential-socket\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.104759 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.103719 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-data\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.104759 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.103748 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-workload-certs\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.104759 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.103797 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-token\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.104759 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.103876 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-envoy\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.104759 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.104000 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istiod-ca-cert\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.104759 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.104123 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-podinfo\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.128113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.117632 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-workload-socket\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.128113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.118335 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-credential-socket\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.128113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.118954 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-workload-certs\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.128113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.119620 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istiod-ca-cert\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.128113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.126702 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-podinfo\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.128113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.127710 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-data\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.129056 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.128997 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-envoy\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.135396 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:35:42.135235 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:35:42.188089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.183907 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd2bl\" (UniqueName: \"kubernetes.io/projected/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-kube-api-access-nd2bl\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.240468 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.226576 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-token\") pod \"e2e-api-a-ca6853-openshift-default-658697f489-nw2dh\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.492710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.492617 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:42.714847 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.714810 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" Nov 03 22:35:42.714847 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.714833 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs" event={"ID":"18e734ea-1f54-4603-9fd9-3261ba6d4ec7","Type":"ContainerDied","Data":"693d906d04ae621102ce255227b927909d3608fa99d812e917b7844e592ca96f"} Nov 03 22:35:42.715415 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.714926 5084 scope.go:122] "RemoveContainer" containerID="751113cfbd3567b732578d94df98a04dd3c74c25941308cf9d2bc41152e1f18c" Nov 03 22:35:42.948468 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.948431 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs"] Nov 03 22:35:42.952666 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.952632 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c211881d-openshift-default-7b56c9b4bf-99ggs"] Nov 03 22:35:42.969968 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:42.969927 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh"] Nov 03 22:35:43.001415 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:35:43.001375 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9a6cfb_6d9f_49ee_a3e6_1a26ca23ab54.slice/crio-79969c7ca07781694eecf99ef45a35e11c2f85181ca24e357b191880cb80a224 WatchSource:0}: Error finding container 79969c7ca07781694eecf99ef45a35e11c2f85181ca24e357b191880cb80a224: Status 404 returned error can't find the container with id 79969c7ca07781694eecf99ef45a35e11c2f85181ca24e357b191880cb80a224 Nov 03 22:35:43.004524 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:43.004480 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:35:43.004657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:43.004559 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:35:43.004657 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:43.004588 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:35:43.728145 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:43.728095 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" event={"ID":"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54","Type":"ContainerStarted","Data":"9ebc673aa0bace8d24191831c86c3abe650f2ddd2cdcb46eb6ec59413e75b6a5"} Nov 03 22:35:43.728627 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:43.728171 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" event={"ID":"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54","Type":"ContainerStarted","Data":"79969c7ca07781694eecf99ef45a35e11c2f85181ca24e357b191880cb80a224"} Nov 03 22:35:43.847622 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:43.847557 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" podStartSLOduration=2.847533576 podStartE2EDuration="2.847533576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:43.845032036 +0000 UTC m=+1711.439064011" watchObservedRunningTime="2026-11-03 22:35:43.847533576 +0000 UTC m=+1711.441565557" Nov 03 22:35:44.493871 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:44.493829 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:44.499662 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:44.499626 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:44.631520 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:44.631484 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e734ea-1f54-4603-9fd9-3261ba6d4ec7" path="/var/lib/kubelet/pods/18e734ea-1f54-4603-9fd9-3261ba6d4ec7/volumes" Nov 03 22:35:44.737891 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:44.737853 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:44.739025 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:44.739003 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:35:50.464287 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:50.464246 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq"] Nov 03 22:35:50.470149 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:50.470112 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:50.587192 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:50.582390 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-f61dc0-openshift-default-dockercfg-4dszb\"" Nov 03 22:35:51.033085 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.032721 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507503d1-913a-4eba-8145-4818fc379376-istiod-ca-cert\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.033397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.033175 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-istio-data\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.033397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.033220 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507503d1-913a-4eba-8145-4818fc379376-istio-token\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.066187 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.033499 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-istio-envoy\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.066187 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.033716 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j6k2\" (UniqueName: \"kubernetes.io/projected/507503d1-913a-4eba-8145-4818fc379376-kube-api-access-2j6k2\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.066187 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.033793 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-workload-certs\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.066187 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.034057 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507503d1-913a-4eba-8145-4818fc379376-istio-podinfo\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.066187 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.034189 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-workload-socket\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.066187 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.034314 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-credential-socket\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.140123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.139453 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507503d1-913a-4eba-8145-4818fc379376-istiod-ca-cert\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.140123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.139546 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-istio-data\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.140123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.139571 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507503d1-913a-4eba-8145-4818fc379376-istio-token\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.140123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.139654 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-istio-envoy\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.140123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.139707 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j6k2\" (UniqueName: \"kubernetes.io/projected/507503d1-913a-4eba-8145-4818fc379376-kube-api-access-2j6k2\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.140123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.139766 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-workload-certs\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.140595 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.140243 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-workload-certs\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.140595 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.140408 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-istio-data\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.140595 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.140463 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507503d1-913a-4eba-8145-4818fc379376-istio-podinfo\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.140595 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.140469 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507503d1-913a-4eba-8145-4818fc379376-istiod-ca-cert\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.140595 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.140574 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-workload-socket\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.140858 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.140671 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-credential-socket\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.141058 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.141033 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-credential-socket\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.141377 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.141349 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-workload-socket\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.141592 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.141574 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq"] Nov 03 22:35:51.144112 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.143108 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-istio-envoy\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.144112 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.143771 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507503d1-913a-4eba-8145-4818fc379376-istio-podinfo\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.200147 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.200106 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507503d1-913a-4eba-8145-4818fc379376-istio-token\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.200307 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.200220 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j6k2\" (UniqueName: \"kubernetes.io/projected/507503d1-913a-4eba-8145-4818fc379376-kube-api-access-2j6k2\") pod \"e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.402156 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.398440 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:51.736148 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.734419 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr"] Nov 03 22:35:51.752099 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.741909 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr" Nov 03 22:35:51.820088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.818002 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ca6853\"" Nov 03 22:35:51.875088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.870349 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5dc5\" (UniqueName: \"kubernetes.io/projected/25ef43d9-40b4-49e3-a8d7-e0d6cc214394-kube-api-access-d5dc5\") pod \"payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr\" (UID: \"25ef43d9-40b4-49e3-a8d7-e0d6cc214394\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr" Nov 03 22:35:51.875088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:51.870498 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ef43d9-40b4-49e3-a8d7-e0d6cc214394-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr\" (UID: \"25ef43d9-40b4-49e3-a8d7-e0d6cc214394\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr" Nov 03 22:35:52.165416 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.020732 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5dc5\" (UniqueName: \"kubernetes.io/projected/25ef43d9-40b4-49e3-a8d7-e0d6cc214394-kube-api-access-d5dc5\") pod \"payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr\" (UID: \"25ef43d9-40b4-49e3-a8d7-e0d6cc214394\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr" Nov 03 22:35:52.165416 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.020851 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ef43d9-40b4-49e3-a8d7-e0d6cc214394-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr\" (UID: \"25ef43d9-40b4-49e3-a8d7-e0d6cc214394\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr" Nov 03 22:35:52.165416 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.022151 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ef43d9-40b4-49e3-a8d7-e0d6cc214394-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr\" (UID: \"25ef43d9-40b4-49e3-a8d7-e0d6cc214394\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr" Nov 03 22:35:52.165416 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.091321 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr"] Nov 03 22:35:52.226854 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.221456 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5dc5\" (UniqueName: \"kubernetes.io/projected/25ef43d9-40b4-49e3-a8d7-e0d6cc214394-kube-api-access-d5dc5\") pod \"payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr\" (UID: \"25ef43d9-40b4-49e3-a8d7-e0d6cc214394\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr" Nov 03 22:35:52.226854 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.226551 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr" Nov 03 22:35:52.240307 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.240269 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb"] Nov 03 22:35:52.252327 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.252289 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb" Nov 03 22:35:52.265721 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.265676 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb"] Nov 03 22:35:52.341100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.332213 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ca6853-dockercfg-lqfd8\"" Nov 03 22:35:52.460468 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.460376 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq"] Nov 03 22:35:52.503758 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:35:52.503724 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507503d1_913a_4eba_8145_4818fc379376.slice/crio-42827eb88ccf6d77078baf53431a61a396bf2bfb444caa47477a0a07d6641762 WatchSource:0}: Error finding container 42827eb88ccf6d77078baf53431a61a396bf2bfb444caa47477a0a07d6641762: Status 404 returned error can't find the container with id 42827eb88ccf6d77078baf53431a61a396bf2bfb444caa47477a0a07d6641762 Nov 03 22:35:52.507332 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.507294 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:35:52.507502 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.507485 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:35:52.507610 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.507596 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:35:52.511360 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.511213 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ac267f-7b6a-4932-8ef1-0b860eef79db-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb\" (UID: \"c9ac267f-7b6a-4932-8ef1-0b860eef79db\") " pod="openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb" Nov 03 22:35:52.511511 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.511471 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrx97\" (UniqueName: \"kubernetes.io/projected/c9ac267f-7b6a-4932-8ef1-0b860eef79db-kube-api-access-hrx97\") pod \"payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb\" (UID: \"c9ac267f-7b6a-4932-8ef1-0b860eef79db\") " pod="openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb" Nov 03 22:35:52.613338 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.613286 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrx97\" (UniqueName: \"kubernetes.io/projected/c9ac267f-7b6a-4932-8ef1-0b860eef79db-kube-api-access-hrx97\") pod \"payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb\" (UID: \"c9ac267f-7b6a-4932-8ef1-0b860eef79db\") " pod="openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb" Nov 03 22:35:52.613550 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.613467 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ac267f-7b6a-4932-8ef1-0b860eef79db-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb\" (UID: \"c9ac267f-7b6a-4932-8ef1-0b860eef79db\") " pod="openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb" Nov 03 22:35:52.614221 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.614196 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ac267f-7b6a-4932-8ef1-0b860eef79db-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb\" (UID: \"c9ac267f-7b6a-4932-8ef1-0b860eef79db\") " pod="openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb" Nov 03 22:35:52.671546 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.671505 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrx97\" (UniqueName: \"kubernetes.io/projected/c9ac267f-7b6a-4932-8ef1-0b860eef79db-kube-api-access-hrx97\") pod \"payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb\" (UID: \"c9ac267f-7b6a-4932-8ef1-0b860eef79db\") " pod="openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb" Nov 03 22:35:52.717709 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.717615 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb" Nov 03 22:35:52.831689 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:52.831644 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr"] Nov 03 22:35:53.117216 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:35:53.117173 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ac267f_7b6a_4932_8ef1_0b860eef79db.slice/crio-6bc14059f23d4092b19ce3aebd94969836fd494db724cf095bf9a32aa5cbf440 WatchSource:0}: Error finding container 6bc14059f23d4092b19ce3aebd94969836fd494db724cf095bf9a32aa5cbf440: Status 404 returned error can't find the container with id 6bc14059f23d4092b19ce3aebd94969836fd494db724cf095bf9a32aa5cbf440 Nov 03 22:35:53.117360 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:53.117275 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb"] Nov 03 22:35:53.254052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:53.253951 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb" event={"ID":"c9ac267f-7b6a-4932-8ef1-0b860eef79db","Type":"ContainerStarted","Data":"a7ec374e8cf6338e5fbe2ff4f2b9683f99b72379d7ef7f9889b9ccc5cd068925"} Nov 03 22:35:53.254052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:53.254021 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb" event={"ID":"c9ac267f-7b6a-4932-8ef1-0b860eef79db","Type":"ContainerStarted","Data":"6bc14059f23d4092b19ce3aebd94969836fd494db724cf095bf9a32aa5cbf440"} Nov 03 22:35:53.254052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:53.254052 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb" Nov 03 22:35:53.256379 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:53.256345 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr" event={"ID":"25ef43d9-40b4-49e3-a8d7-e0d6cc214394","Type":"ContainerStarted","Data":"28cfc8445d395200bc36e3abee7827d45cbc749bc92d36256502821ec5df836a"} Nov 03 22:35:53.256531 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:53.256441 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr" event={"ID":"25ef43d9-40b4-49e3-a8d7-e0d6cc214394","Type":"ContainerStarted","Data":"a1f1b29a5bfdf1215aced355e348db1658a79387d01e0f3463f3a30f8e5c46e3"} Nov 03 22:35:53.256531 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:53.256466 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr" Nov 03 22:35:53.258341 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:53.258307 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" event={"ID":"507503d1-913a-4eba-8145-4818fc379376","Type":"ContainerStarted","Data":"3da55bf7ce31e3848f6d4c61d2ff1fd075642f74d63ad136b9266400a53b7c7d"} Nov 03 22:35:53.258459 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:53.258353 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" event={"ID":"507503d1-913a-4eba-8145-4818fc379376","Type":"ContainerStarted","Data":"42827eb88ccf6d77078baf53431a61a396bf2bfb444caa47477a0a07d6641762"} Nov 03 22:35:53.373967 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:53.373891 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb" podStartSLOduration=2.3738705270000002 podStartE2EDuration="2.373870527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:53.369229648 +0000 UTC m=+1720.963261651" watchObservedRunningTime="2026-11-03 22:35:53.373870527 +0000 UTC m=+1720.967902502" Nov 03 22:35:53.399290 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:53.399253 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:53.405281 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:53.405245 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:53.510227 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:53.504647 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" podStartSLOduration=3.504621973 podStartE2EDuration="3.504621973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:53.487782525 +0000 UTC m=+1721.081814500" watchObservedRunningTime="2026-11-03 22:35:53.504621973 +0000 UTC m=+1721.098653950" Nov 03 22:35:53.628628 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:53.610827 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr" podStartSLOduration=2.610806168 podStartE2EDuration="2.610806168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:53.609490014 +0000 UTC m=+1721.203521988" watchObservedRunningTime="2026-11-03 22:35:53.610806168 +0000 UTC m=+1721.204838142" Nov 03 22:35:54.269188 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.269141 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl"] Nov 03 22:35:54.274831 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.274790 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:54.275002 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.274874 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:35:54.275002 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.274943 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:35:54.326596 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.326554 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl"] Nov 03 22:35:54.357632 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.357592 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76021af-474f-4be2-bafa-2eeb78f1144a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl\" (UID: \"d76021af-474f-4be2-bafa-2eeb78f1144a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:35:54.357914 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.357893 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d76021af-474f-4be2-bafa-2eeb78f1144a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl\" (UID: \"d76021af-474f-4be2-bafa-2eeb78f1144a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:35:54.358121 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.358103 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76021af-474f-4be2-bafa-2eeb78f1144a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl\" (UID: \"d76021af-474f-4be2-bafa-2eeb78f1144a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:35:54.358522 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.358491 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvxjj\" (UniqueName: \"kubernetes.io/projected/d76021af-474f-4be2-bafa-2eeb78f1144a-kube-api-access-lvxjj\") pod \"maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl\" (UID: \"d76021af-474f-4be2-bafa-2eeb78f1144a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:35:54.372098 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.372046 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-37b70b-serving-cert\"" Nov 03 22:35:54.459914 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.459867 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76021af-474f-4be2-bafa-2eeb78f1144a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl\" (UID: \"d76021af-474f-4be2-bafa-2eeb78f1144a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:35:54.460205 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.459936 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d76021af-474f-4be2-bafa-2eeb78f1144a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl\" (UID: \"d76021af-474f-4be2-bafa-2eeb78f1144a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:35:54.460205 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.459961 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76021af-474f-4be2-bafa-2eeb78f1144a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl\" (UID: \"d76021af-474f-4be2-bafa-2eeb78f1144a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:35:54.460205 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.460191 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvxjj\" (UniqueName: \"kubernetes.io/projected/d76021af-474f-4be2-bafa-2eeb78f1144a-kube-api-access-lvxjj\") pod \"maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl\" (UID: \"d76021af-474f-4be2-bafa-2eeb78f1144a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:35:54.461308 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.460747 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76021af-474f-4be2-bafa-2eeb78f1144a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl\" (UID: \"d76021af-474f-4be2-bafa-2eeb78f1144a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:35:54.461308 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:35:54.461286 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-serving-cert: secret "maas-api-e2e-shared-a-w4-37b70b-serving-cert" not found Nov 03 22:35:54.461629 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:35:54.461354 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d76021af-474f-4be2-bafa-2eeb78f1144a-maas-api-tls podName:d76021af-474f-4be2-bafa-2eeb78f1144a nodeName:}" failed. No retries permitted until 2026-11-03 22:35:54.961332423 +0000 UTC m=+1722.555364376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d76021af-474f-4be2-bafa-2eeb78f1144a-maas-api-tls") pod "maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" (UID: "d76021af-474f-4be2-bafa-2eeb78f1144a") : secret "maas-api-e2e-shared-a-w4-37b70b-serving-cert" not found Nov 03 22:35:54.479723 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.479680 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d76021af-474f-4be2-bafa-2eeb78f1144a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl\" (UID: \"d76021af-474f-4be2-bafa-2eeb78f1144a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:35:54.518467 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.518433 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvxjj\" (UniqueName: \"kubernetes.io/projected/d76021af-474f-4be2-bafa-2eeb78f1144a-kube-api-access-lvxjj\") pod \"maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl\" (UID: \"d76021af-474f-4be2-bafa-2eeb78f1144a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:35:54.974344 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.974297 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76021af-474f-4be2-bafa-2eeb78f1144a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl\" (UID: \"d76021af-474f-4be2-bafa-2eeb78f1144a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:35:54.977672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:54.977635 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76021af-474f-4be2-bafa-2eeb78f1144a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl\" (UID: \"d76021af-474f-4be2-bafa-2eeb78f1144a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:35:55.191809 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:55.191754 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:35:55.628725 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:55.628689 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl"] Nov 03 22:35:55.629534 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:35:55.629501 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76021af_474f_4be2_bafa_2eeb78f1144a.slice/crio-4e86e7f5a3c34a261e562bd62e444daf1cbfe0f93f6808efe80e9c7a017bf5b5 WatchSource:0}: Error finding container 4e86e7f5a3c34a261e562bd62e444daf1cbfe0f93f6808efe80e9c7a017bf5b5: Status 404 returned error can't find the container with id 4e86e7f5a3c34a261e562bd62e444daf1cbfe0f93f6808efe80e9c7a017bf5b5 Nov 03 22:35:55.631234 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:55.631213 5084 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:35:56.295663 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:56.295613 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" event={"ID":"d76021af-474f-4be2-bafa-2eeb78f1144a","Type":"ContainerStarted","Data":"4e86e7f5a3c34a261e562bd62e444daf1cbfe0f93f6808efe80e9c7a017bf5b5"} Nov 03 22:35:57.364093 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:57.354090 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" event={"ID":"d76021af-474f-4be2-bafa-2eeb78f1144a","Type":"ContainerStarted","Data":"3eaaead41050c8f8aef736b9a03678cf60f9485d0daf83054cdb9c3b012a71f7"} Nov 03 22:35:57.364093 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:57.358554 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:35:57.488180 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:35:57.482608 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" podStartSLOduration=2.532305296 podStartE2EDuration="3.482588003s" totalImagesPullingTime="950.282707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:57.479549334 +0000 UTC m=+1725.073581309" watchObservedRunningTime="2026-11-03 22:35:57.482588003 +0000 UTC m=+1725.076619977" Nov 03 22:36:03.370625 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:03.370592 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:36:04.277469 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:04.277429 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb" Nov 03 22:36:04.294354 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:04.283621 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr" Nov 03 22:36:10.047106 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:10.041484 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-28lxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:36:10.047106 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:10.041598 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" podUID="d3d5085e-0b4b-46a0-83cc-339d5550dd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:36:12.027006 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:36:12.026959 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:36:15.969274 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:15.969241 5084 scope.go:122] "RemoveContainer" containerID="908a8146a87c0a0ea0543e29b02f8ccfa7f6c3565c09785acbf1e83b1f909589" Nov 03 22:36:15.982936 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:15.982909 5084 scope.go:122] "RemoveContainer" containerID="157889b2a5bcdb2edaaee2205de38e62bed07c78d14ea89cf8380dbafaabddc5" Nov 03 22:36:15.996013 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:15.995975 5084 scope.go:122] "RemoveContainer" containerID="69de2401c341590480f27fed4448dfbb8adf5b19a57c05bfaad71e6d8fbedf43" Nov 03 22:36:16.007705 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:16.007663 5084 scope.go:122] "RemoveContainer" containerID="cbad2739761b20e7b4f4847cc365575d3e4af0660ef27d29a6fb6dbeea1b3cee" Nov 03 22:36:16.019877 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:16.019849 5084 scope.go:122] "RemoveContainer" containerID="7d8d0f28fabd04272c8a29893d6a14b1d7bc2df80207fcc7b9d6f59166bcea59" Nov 03 22:36:36.847582 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:36.847544 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9"] Nov 03 22:36:36.852448 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:36.852417 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:36:36.904955 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:36.904915 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9"] Nov 03 22:36:36.950515 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:36.950477 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ca6853-serving-cert\"" Nov 03 22:36:36.965806 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:36.965766 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-metrics-certs\") pod \"maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9\" (UID: \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:36:36.966022 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:36.965931 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-openshift-service-ca\") pod \"maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9\" (UID: \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:36:36.966022 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:36.965980 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-maas-api-tls\") pod \"maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9\" (UID: \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:36:36.966202 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:36.966084 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlbp9\" (UniqueName: \"kubernetes.io/projected/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-kube-api-access-zlbp9\") pod \"maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9\" (UID: \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:36:37.067975 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:37.067922 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlbp9\" (UniqueName: \"kubernetes.io/projected/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-kube-api-access-zlbp9\") pod \"maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9\" (UID: \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:36:37.068225 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:37.068160 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-metrics-certs\") pod \"maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9\" (UID: \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:36:37.068310 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:37.068273 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-openshift-service-ca\") pod \"maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9\" (UID: \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:36:37.068365 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:37.068310 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-maas-api-tls\") pod \"maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9\" (UID: \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:36:37.068487 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:36:37.068471 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-serving-cert: secret "maas-api-e2e-api-a-ca6853-serving-cert" not found Nov 03 22:36:37.068560 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:36:37.068537 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-maas-api-tls podName:dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20 nodeName:}" failed. No retries permitted until 2026-11-03 22:36:37.568515336 +0000 UTC m=+1765.162547291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-maas-api-tls") pod "maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" (UID: "dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20") : secret "maas-api-e2e-api-a-ca6853-serving-cert" not found Nov 03 22:36:37.069094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:37.069037 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-openshift-service-ca\") pod \"maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9\" (UID: \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:36:37.070957 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:37.070931 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-metrics-certs\") pod \"maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9\" (UID: \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:36:37.134577 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:37.134240 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlbp9\" (UniqueName: \"kubernetes.io/projected/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-kube-api-access-zlbp9\") pod \"maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9\" (UID: \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:36:37.588500 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:37.588453 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-maas-api-tls\") pod \"maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9\" (UID: \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:36:37.591163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:37.591131 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-maas-api-tls\") pod \"maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9\" (UID: \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:36:37.771819 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:37.771772 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:36:38.160809 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:36:38.160776 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4621b2_6d1d_4ff2_a82e_d1b7d2285f20.slice/crio-f4286b9f369a8a356b7fcc5a9a3a57a6bd714b680849156d29c05f6df5054f94 WatchSource:0}: Error finding container f4286b9f369a8a356b7fcc5a9a3a57a6bd714b680849156d29c05f6df5054f94: Status 404 returned error can't find the container with id f4286b9f369a8a356b7fcc5a9a3a57a6bd714b680849156d29c05f6df5054f94 Nov 03 22:36:38.161195 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:38.161091 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9"] Nov 03 22:36:38.976464 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:38.976415 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" event={"ID":"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20","Type":"ContainerStarted","Data":"f4286b9f369a8a356b7fcc5a9a3a57a6bd714b680849156d29c05f6df5054f94"} Nov 03 22:36:39.920680 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:39.918795 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r"] Nov 03 22:36:39.922689 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:39.922637 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" podUID="4324c92c-c67b-4407-868b-eecc823ac436" containerName="istio-proxy" containerID="cri-o://114c46a3f6d4e6e3ea4a47e0ce33fffea4cb55a37f860342ad2184b37fec5189" gracePeriod=30 Nov 03 22:36:39.992406 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:39.992360 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" event={"ID":"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20","Type":"ContainerStarted","Data":"136cfa35edd88865aacc778b00737bb5a9ece354a12ce60fd5704c5046374ef7"} Nov 03 22:36:39.993401 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:39.993372 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:36:40.169272 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:40.169215 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" podStartSLOduration=3.196040798 podStartE2EDuration="4.169198879s" totalImagesPullingTime="973.158081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:40.168228734 +0000 UTC m=+1767.762260708" watchObservedRunningTime="2026-11-03 22:36:40.169198879 +0000 UTC m=+1767.763230852" Nov 03 22:36:42.026715 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:36:42.026671 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:36:45.855359 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.855322 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:36:45.860365 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.860312 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-credential-socket\") pod \"4324c92c-c67b-4407-868b-eecc823ac436\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " Nov 03 22:36:45.860545 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.860396 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-istio-envoy\") pod \"4324c92c-c67b-4407-868b-eecc823ac436\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " Nov 03 22:36:45.860545 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.860440 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-workload-socket\") pod \"4324c92c-c67b-4407-868b-eecc823ac436\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " Nov 03 22:36:45.860545 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.860486 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4324c92c-c67b-4407-868b-eecc823ac436-kube-api-access-kk7jj\" (UniqueName: \"kubernetes.io/projected/4324c92c-c67b-4407-868b-eecc823ac436-kube-api-access-kk7jj\") pod \"4324c92c-c67b-4407-868b-eecc823ac436\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " Nov 03 22:36:45.860545 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.860520 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4324c92c-c67b-4407-868b-eecc823ac436-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4324c92c-c67b-4407-868b-eecc823ac436-istio-podinfo\") pod \"4324c92c-c67b-4407-868b-eecc823ac436\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " Nov 03 22:36:45.860708 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.860565 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4324c92c-c67b-4407-868b-eecc823ac436-istio-token\" (UniqueName: \"kubernetes.io/projected/4324c92c-c67b-4407-868b-eecc823ac436-istio-token\") pod \"4324c92c-c67b-4407-868b-eecc823ac436\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " Nov 03 22:36:45.860708 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.860598 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-istio-data\") pod \"4324c92c-c67b-4407-868b-eecc823ac436\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " Nov 03 22:36:45.860708 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.860656 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-credential-socket" pod "4324c92c-c67b-4407-868b-eecc823ac436" (UID: "4324c92c-c67b-4407-868b-eecc823ac436"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:45.860921 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.860903 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-workload-certs\") pod \"4324c92c-c67b-4407-868b-eecc823ac436\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " Nov 03 22:36:45.860976 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.860958 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4324c92c-c67b-4407-868b-eecc823ac436-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4324c92c-c67b-4407-868b-eecc823ac436-istiod-ca-cert\") pod \"4324c92c-c67b-4407-868b-eecc823ac436\" (UID: \"4324c92c-c67b-4407-868b-eecc823ac436\") " Nov 03 22:36:45.861322 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.861296 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-workload-socket" pod "4324c92c-c67b-4407-868b-eecc823ac436" (UID: "4324c92c-c67b-4407-868b-eecc823ac436"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:45.861847 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.861818 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:45.861847 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.861833 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4324c92c-c67b-4407-868b-eecc823ac436-istiod-ca-cert" pod "4324c92c-c67b-4407-868b-eecc823ac436" (UID: "4324c92c-c67b-4407-868b-eecc823ac436"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:45.862039 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.861848 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:45.862268 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.862244 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-istio-data" pod "4324c92c-c67b-4407-868b-eecc823ac436" (UID: "4324c92c-c67b-4407-868b-eecc823ac436"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:45.862466 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.862433 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-workload-certs" pod "4324c92c-c67b-4407-868b-eecc823ac436" (UID: "4324c92c-c67b-4407-868b-eecc823ac436"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:45.864669 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.864638 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4324c92c-c67b-4407-868b-eecc823ac436-kube-api-access-kk7jj" pod "4324c92c-c67b-4407-868b-eecc823ac436" (UID: "4324c92c-c67b-4407-868b-eecc823ac436"). InnerVolumeSpecName "kube-api-access-kk7jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:45.865222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.865193 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4324c92c-c67b-4407-868b-eecc823ac436-istio-podinfo" pod "4324c92c-c67b-4407-868b-eecc823ac436" (UID: "4324c92c-c67b-4407-868b-eecc823ac436"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:45.865222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.865198 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4324c92c-c67b-4407-868b-eecc823ac436-istio-token" pod "4324c92c-c67b-4407-868b-eecc823ac436" (UID: "4324c92c-c67b-4407-868b-eecc823ac436"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:45.866692 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.866573 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-istio-envoy" pod "4324c92c-c67b-4407-868b-eecc823ac436" (UID: "4324c92c-c67b-4407-868b-eecc823ac436"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:45.963561 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.963523 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4324c92c-c67b-4407-868b-eecc823ac436-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:45.963561 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.963563 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:45.963784 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.963579 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:45.963784 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.963594 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4324c92c-c67b-4407-868b-eecc823ac436-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:45.963784 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.963608 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4324c92c-c67b-4407-868b-eecc823ac436-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:45.963784 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.963623 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk7jj\" (UniqueName: \"kubernetes.io/projected/4324c92c-c67b-4407-868b-eecc823ac436-kube-api-access-kk7jj\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:45.963784 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:45.963637 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4324c92c-c67b-4407-868b-eecc823ac436-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:36:46.342294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.342254 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw"] Nov 03 22:36:46.343554 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.343523 5084 cpu_manager.go:414] "containerMap: removing container" podUID="4324c92c-c67b-4407-868b-eecc823ac436" containerName="istio-proxy" Nov 03 22:36:46.343554 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.343551 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4324c92c-c67b-4407-868b-eecc823ac436" containerName="istio-proxy" Nov 03 22:36:46.343756 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.343715 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="4324c92c-c67b-4407-868b-eecc823ac436" containerName="istio-proxy" Nov 03 22:36:46.348052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.348027 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:36:46.371232 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.371182 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw\" (UID: \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:36:46.371462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.371439 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw\" (UID: \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:36:46.371580 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.371560 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhqqx\" (UniqueName: \"kubernetes.io/projected/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-kube-api-access-hhqqx\") pod \"maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw\" (UID: \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:36:46.371716 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.371692 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw\" (UID: \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:36:46.394743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.394701 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw"] Nov 03 22:36:46.401384 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.401353 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-f61dc0-serving-cert\"" Nov 03 22:36:46.415502 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.415461 5084 generic.go:356] "Generic (PLEG): container finished" podID="4324c92c-c67b-4407-868b-eecc823ac436" containerID="114c46a3f6d4e6e3ea4a47e0ce33fffea4cb55a37f860342ad2184b37fec5189" exitCode=0 Nov 03 22:36:46.415720 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.415522 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" Nov 03 22:36:46.415720 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.415541 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" event={"ID":"4324c92c-c67b-4407-868b-eecc823ac436","Type":"ContainerDied","Data":"114c46a3f6d4e6e3ea4a47e0ce33fffea4cb55a37f860342ad2184b37fec5189"} Nov 03 22:36:46.415720 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.415614 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r" event={"ID":"4324c92c-c67b-4407-868b-eecc823ac436","Type":"ContainerDied","Data":"318570719d14aa0bd8cce7c36f296f556b606e4ac2648a72e18259bd8ebd47b2"} Nov 03 22:36:46.415720 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.415644 5084 scope.go:122] "RemoveContainer" containerID="114c46a3f6d4e6e3ea4a47e0ce33fffea4cb55a37f860342ad2184b37fec5189" Nov 03 22:36:46.428600 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.428575 5084 scope.go:122] "RemoveContainer" containerID="114c46a3f6d4e6e3ea4a47e0ce33fffea4cb55a37f860342ad2184b37fec5189" Nov 03 22:36:46.428949 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:36:46.428926 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114c46a3f6d4e6e3ea4a47e0ce33fffea4cb55a37f860342ad2184b37fec5189\": container with ID starting with 114c46a3f6d4e6e3ea4a47e0ce33fffea4cb55a37f860342ad2184b37fec5189 not found: ID does not exist" containerID="114c46a3f6d4e6e3ea4a47e0ce33fffea4cb55a37f860342ad2184b37fec5189" Nov 03 22:36:46.429002 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.428959 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114c46a3f6d4e6e3ea4a47e0ce33fffea4cb55a37f860342ad2184b37fec5189"} err="failed to get container status \"114c46a3f6d4e6e3ea4a47e0ce33fffea4cb55a37f860342ad2184b37fec5189\": rpc error: code = NotFound desc = could not find container \"114c46a3f6d4e6e3ea4a47e0ce33fffea4cb55a37f860342ad2184b37fec5189\": container with ID starting with 114c46a3f6d4e6e3ea4a47e0ce33fffea4cb55a37f860342ad2184b37fec5189 not found: ID does not exist" Nov 03 22:36:46.474124 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.474023 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw\" (UID: \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:36:46.474335 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.474158 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhqqx\" (UniqueName: \"kubernetes.io/projected/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-kube-api-access-hhqqx\") pod \"maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw\" (UID: \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:36:46.474335 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.474190 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw\" (UID: \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:36:46.474469 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.474343 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw\" (UID: \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:36:46.474522 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:36:46.474477 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-serving-cert: secret "maas-api-e2e-shared-a-w3-f61dc0-serving-cert" not found Nov 03 22:36:46.474577 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:36:46.474563 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-maas-api-tls podName:1995b4ad-49a3-4ddd-b04f-c8b02e59cf35 nodeName:}" failed. No retries permitted until 2026-11-03 22:36:46.974536869 +0000 UTC m=+1774.568568837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-maas-api-tls") pod "maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" (UID: "1995b4ad-49a3-4ddd-b04f-c8b02e59cf35") : secret "maas-api-e2e-shared-a-w3-f61dc0-serving-cert" not found Nov 03 22:36:46.474945 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.474913 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw\" (UID: \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:36:46.477263 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.477236 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw\" (UID: \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:36:46.533761 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.533711 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhqqx\" (UniqueName: \"kubernetes.io/projected/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-kube-api-access-hhqqx\") pod \"maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw\" (UID: \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:36:46.597293 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.597136 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r"] Nov 03 22:36:46.621244 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.620432 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-7dw6r"] Nov 03 22:36:46.683961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:46.683899 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4324c92c-c67b-4407-868b-eecc823ac436" path="/var/lib/kubelet/pods/4324c92c-c67b-4407-868b-eecc823ac436/volumes" Nov 03 22:36:47.018772 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:47.018736 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw\" (UID: \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:36:47.020390 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:47.020361 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:36:47.023349 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:47.023321 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw\" (UID: \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:36:47.265408 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:47.265368 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:36:47.665339 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:47.665304 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw"] Nov 03 22:36:47.666918 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:36:47.666887 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1995b4ad_49a3_4ddd_b04f_c8b02e59cf35.slice/crio-dad2cd82480531d9e639b631ffbc45ab86646f30f0a9e012e013aacdf199b6d9 WatchSource:0}: Error finding container dad2cd82480531d9e639b631ffbc45ab86646f30f0a9e012e013aacdf199b6d9: Status 404 returned error can't find the container with id dad2cd82480531d9e639b631ffbc45ab86646f30f0a9e012e013aacdf199b6d9 Nov 03 22:36:48.508977 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:48.507976 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" event={"ID":"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35","Type":"ContainerStarted","Data":"dad2cd82480531d9e639b631ffbc45ab86646f30f0a9e012e013aacdf199b6d9"} Nov 03 22:36:49.572149 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:49.572102 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" event={"ID":"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35","Type":"ContainerStarted","Data":"084e9e4cef95965b632466a98a4ee91929d86c059c48abc68a802b81d6104060"} Nov 03 22:36:49.573455 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:49.573430 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:36:49.695473 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:49.695400 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" podStartSLOduration=2.690372546 podStartE2EDuration="3.695375941s" totalImagesPullingTime="1.005003395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:49.690029416 +0000 UTC m=+1777.284061392" watchObservedRunningTime="2026-11-03 22:36:49.695375941 +0000 UTC m=+1777.289407917" Nov 03 22:36:56.291620 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:56.291575 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2"] Nov 03 22:36:56.292130 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:56.291899 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" podUID="5be671e7-45d4-4e28-b384-2f3dd6397ecb" containerName="istio-proxy" containerID="cri-o://69842cadac3a1434e1507d203187c9f4d9e1c3658fae8953789d6d276467e1bd" gracePeriod=30 Nov 03 22:36:56.732024 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:36:56.731943 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:37:00.743924 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:00.743870 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:37:00.757370 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:00.744687 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" podUID="5be671e7-45d4-4e28-b384-2f3dd6397ecb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:37:02.279776 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:02.260567 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:02.821956 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:02.817636 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:37:03.315382 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.297907 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:03.316860 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.316804 5084 generic.go:356] "Generic (PLEG): container finished" podID="5be671e7-45d4-4e28-b384-2f3dd6397ecb" containerID="69842cadac3a1434e1507d203187c9f4d9e1c3658fae8953789d6d276467e1bd" exitCode=0 Nov 03 22:37:03.317628 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.317602 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" Nov 03 22:37:03.335685 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.335652 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:37:03.337472 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.335985 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" event={"ID":"5be671e7-45d4-4e28-b384-2f3dd6397ecb","Type":"ContainerDied","Data":"69842cadac3a1434e1507d203187c9f4d9e1c3658fae8953789d6d276467e1bd"} Nov 03 22:37:03.368220 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.344118 5084 scope.go:122] "RemoveContainer" containerID="69842cadac3a1434e1507d203187c9f4d9e1c3658fae8953789d6d276467e1bd" Nov 03 22:37:03.387611 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.386093 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:03.412152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.411666 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" podUID="3b3927af-60a2-4432-99d6-b999a96f315f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 22:37:03.423141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.412583 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be671e7-45d4-4e28-b384-2f3dd6397ecb-kube-api-access-hdvh8\" (UniqueName: \"kubernetes.io/projected/5be671e7-45d4-4e28-b384-2f3dd6397ecb-kube-api-access-hdvh8\") pod \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " Nov 03 22:37:03.423141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.412671 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-envoy\") pod \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " Nov 03 22:37:03.423141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.412767 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istiod-ca-cert\") pod \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " Nov 03 22:37:03.423141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.412905 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-data\") pod \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " Nov 03 22:37:03.423141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.412955 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-token\" (UniqueName: \"kubernetes.io/projected/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-token\") pod \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " Nov 03 22:37:03.423141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.412993 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-podinfo\") pod \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " Nov 03 22:37:03.423141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.413113 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-workload-socket\") pod \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " Nov 03 22:37:03.423141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.413174 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-credential-socket\") pod \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " Nov 03 22:37:03.423141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.413253 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-workload-certs\") pod \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\" (UID: \"5be671e7-45d4-4e28-b384-2f3dd6397ecb\") " Nov 03 22:37:03.423141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.414397 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-workload-certs" pod "5be671e7-45d4-4e28-b384-2f3dd6397ecb" (UID: "5be671e7-45d4-4e28-b384-2f3dd6397ecb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:03.569541 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.569415 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-credential-socket" pod "5be671e7-45d4-4e28-b384-2f3dd6397ecb" (UID: "5be671e7-45d4-4e28-b384-2f3dd6397ecb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:03.574104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.573425 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-data" pod "5be671e7-45d4-4e28-b384-2f3dd6397ecb" (UID: "5be671e7-45d4-4e28-b384-2f3dd6397ecb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:03.577345 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.577299 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istiod-ca-cert" pod "5be671e7-45d4-4e28-b384-2f3dd6397ecb" (UID: "5be671e7-45d4-4e28-b384-2f3dd6397ecb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:03.619102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.617106 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:03.619102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.617149 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:03.619102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.617166 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:03.619102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.617180 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:03.687646 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.661764 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 22:37:03.930818 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.889749 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-podinfo" pod "5be671e7-45d4-4e28-b384-2f3dd6397ecb" (UID: "5be671e7-45d4-4e28-b384-2f3dd6397ecb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:37:03.930818 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.898103 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:03.930818 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.929291 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-workload-socket" pod "5be671e7-45d4-4e28-b384-2f3dd6397ecb" (UID: "5be671e7-45d4-4e28-b384-2f3dd6397ecb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:04.131091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.035262 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:04.131091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:03.970783 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-token" pod "5be671e7-45d4-4e28-b384-2f3dd6397ecb" (UID: "5be671e7-45d4-4e28-b384-2f3dd6397ecb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:04.131506 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.131461 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" podUID="bdf09e52-d50f-4d0e-812c-60220d5533b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.132:8000/ready\": context deadline exceeded" Nov 03 22:37:04.160520 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.156652 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-envoy" pod "5be671e7-45d4-4e28-b384-2f3dd6397ecb" (UID: "5be671e7-45d4-4e28-b384-2f3dd6397ecb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:04.160520 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.156858 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be671e7-45d4-4e28-b384-2f3dd6397ecb-kube-api-access-hdvh8" pod "5be671e7-45d4-4e28-b384-2f3dd6397ecb" (UID: "5be671e7-45d4-4e28-b384-2f3dd6397ecb"). InnerVolumeSpecName "kube-api-access-hdvh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:04.238047 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.236227 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:04.238047 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.236274 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdvh8\" (UniqueName: \"kubernetes.io/projected/5be671e7-45d4-4e28-b384-2f3dd6397ecb-kube-api-access-hdvh8\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:04.238047 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.236292 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:04.238047 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.236307 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5be671e7-45d4-4e28-b384-2f3dd6397ecb-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:04.306582 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.298320 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:04.394106 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.388189 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:04.394106 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.389022 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:37:04.415145 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.412707 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" podUID="3b3927af-60a2-4432-99d6-b999a96f315f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:04.417274 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.416224 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:37:04.427051 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.426979 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2" event={"ID":"5be671e7-45d4-4e28-b384-2f3dd6397ecb","Type":"ContainerDied","Data":"32c9cf2e07671b6a409059112d462b415e5371a46b67b6263ab85bc2ad37d7a1"} Nov 03 22:37:04.558345 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.558294 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2"] Nov 03 22:37:04.622087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.622019 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-btxm2"] Nov 03 22:37:04.688768 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.688713 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 22:37:04.753256 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.742854 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5be671e7-45d4-4e28-b384-2f3dd6397ecb" path="/var/lib/kubelet/pods/5be671e7-45d4-4e28-b384-2f3dd6397ecb/volumes" Nov 03 22:37:04.767655 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:04.767617 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:37:05.121084 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:05.115507 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 22:37:05.121084 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:05.116340 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:37:05.179087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:05.176335 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" podUID="bdf09e52-d50f-4d0e-812c-60220d5533b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:05.179087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:05.176961 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:37:05.307480 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:05.306207 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:05.349448 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:05.349417 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:37:05.432177 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:05.426691 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" podUID="3b3927af-60a2-4432-99d6-b999a96f315f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 22:37:05.441122 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:05.440218 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:05.545249 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:05.545199 5084 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:05.693319 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:05.693204 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:05.752236 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:05.752193 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:37:06.185320 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:06.185266 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" podUID="bdf09e52-d50f-4d0e-812c-60220d5533b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:06.345292 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:06.344507 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:37:06.370091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:06.356220 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:06.452645 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:06.448925 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": context deadline exceeded" Nov 03 22:37:06.598721 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:06.598685 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:37:07.003082 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:07.003028 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:37:07.003870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:07.003848 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:37:08.370089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:08.359183 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:08.370089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:08.368170 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" podUID="3b3927af-60a2-4432-99d6-b999a96f315f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:08.506094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:08.503167 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:10.375302 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:10.375254 5084 patch_prober.go:29] interesting pod/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:37:10.375818 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:10.375340 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" podUID="9477ee77-a2d4-4500-9bf8-5f2b8f363928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:37:12.033689 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:37:12.033633 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:37:12.954458 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:12.954412 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:37:12.960243 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:12.960209 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:37:14.137555 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:14.137511 5084 generic.go:356] "Generic (PLEG): container finished" podID="2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" containerID="9ebc673aa0bace8d24191831c86c3abe650f2ddd2cdcb46eb6ec59413e75b6a5" exitCode=0 Nov 03 22:37:14.138147 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:14.137616 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" event={"ID":"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54","Type":"ContainerDied","Data":"9ebc673aa0bace8d24191831c86c3abe650f2ddd2cdcb46eb6ec59413e75b6a5"} Nov 03 22:37:14.138245 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:14.138215 5084 scope.go:122] "RemoveContainer" containerID="9ebc673aa0bace8d24191831c86c3abe650f2ddd2cdcb46eb6ec59413e75b6a5" Nov 03 22:37:14.140232 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:14.140194 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:14.140349 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:14.140235 5084 generic.go:356] "Generic (PLEG): container finished" podID="9477ee77-a2d4-4500-9bf8-5f2b8f363928" containerID="2eea585f1102ee82ddc0cf10f595754f4b3e0bb1a0266baed94d81193649add7" exitCode=0 Nov 03 22:37:14.140349 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:14.140274 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:14.140349 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:14.140307 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" event={"ID":"9477ee77-a2d4-4500-9bf8-5f2b8f363928","Type":"ContainerDied","Data":"2eea585f1102ee82ddc0cf10f595754f4b3e0bb1a0266baed94d81193649add7"} Nov 03 22:37:14.140349 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:14.140332 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:14.140866 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:14.140814 5084 scope.go:122] "RemoveContainer" containerID="2eea585f1102ee82ddc0cf10f595754f4b3e0bb1a0266baed94d81193649add7" Nov 03 22:37:14.142532 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:14.142500 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:14.142639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:14.142581 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:14.142706 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:14.142634 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:37:14.738384 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:14.738346 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:37:15.151236 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:15.151203 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" event={"ID":"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54","Type":"ContainerStarted","Data":"cab8ecf5f16bd477ba766c564947427b40f10590b34619e75b8eb6276bfa0a72"} Nov 03 22:37:15.153502 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:15.153474 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" event={"ID":"9477ee77-a2d4-4500-9bf8-5f2b8f363928","Type":"ContainerStarted","Data":"c42fd10620bd2d31a41d6eadfff13289089085f4305cc0268af6392226b128a8"} Nov 03 22:37:15.310086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:15.290452 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:37:15.310086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:15.290588 5084 patch_prober.go:29] interesting pod/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:37:15.310086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:15.290646 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" podUID="9477ee77-a2d4-4500-9bf8-5f2b8f363928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 22:37:15.498504 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:15.497496 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:37:15.505109 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:15.504053 5084 patch_prober.go:29] interesting pod/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 22:37:15.505109 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:15.504147 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" podUID="2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 22:37:16.118088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:16.117856 5084 scope.go:122] "RemoveContainer" containerID="3e531aa1f16ef0dc6c44dd31b6b3336c27a1f78974f02b85ef708d33071a4c09" Nov 03 22:37:16.285171 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:16.276103 5084 patch_prober.go:29] interesting pod/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:37:16.285171 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:16.276200 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" podUID="9477ee77-a2d4-4500-9bf8-5f2b8f363928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 22:37:16.499154 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:16.495552 5084 patch_prober.go:29] interesting pod/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 22:37:16.499154 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:16.495619 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" podUID="2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 22:37:17.280090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:17.279245 5084 patch_prober.go:29] interesting pod/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:37:17.280090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:17.279320 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" podUID="9477ee77-a2d4-4500-9bf8-5f2b8f363928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 22:37:17.502086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:17.495533 5084 patch_prober.go:29] interesting pod/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 22:37:17.502086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:17.495598 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" podUID="2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 22:37:18.272517 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:18.272477 5084 patch_prober.go:29] interesting pod/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:37:18.272895 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:18.272859 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" podUID="9477ee77-a2d4-4500-9bf8-5f2b8f363928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 22:37:18.500092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:18.497012 5084 patch_prober.go:29] interesting pod/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 22:37:18.500092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:18.497099 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" podUID="2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 22:37:19.277097 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:19.276572 5084 patch_prober.go:29] interesting pod/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:37:19.277097 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:19.276640 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" podUID="9477ee77-a2d4-4500-9bf8-5f2b8f363928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 22:37:19.496087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:19.495523 5084 patch_prober.go:29] interesting pod/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 22:37:19.496087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:19.495596 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" podUID="2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 22:37:20.329090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:20.326029 5084 patch_prober.go:29] interesting pod/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 22:37:20.329090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:20.326112 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" podUID="9477ee77-a2d4-4500-9bf8-5f2b8f363928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 22:37:20.505090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:20.499514 5084 patch_prober.go:29] interesting pod/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 22:37:20.505090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:20.499603 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" podUID="2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 22:37:21.275650 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:21.275614 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:37:21.276116 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:21.276084 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:37:21.277179 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:21.277158 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:37:21.499807 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:21.499775 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:37:21.500257 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:21.500098 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:37:21.501013 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:21.500989 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:37:30.292024 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:30.291968 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4"] Nov 03 22:37:30.415692 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:30.415648 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v"] Nov 03 22:37:35.293417 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:35.293371 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4" podUID="ccd4b6d3-6913-4191-ac28-028701a386e8" containerName="payload-processing" containerID="cri-o://0091922fcb609de295e6b47565f9e9d9a350642629a2c13ab4e06d8941c82dcf" gracePeriod=25 Nov 03 22:37:35.416585 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:35.416540 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v" podUID="fae1546c-f272-4c56-8d56-5933d34c703b" containerName="payload-pre-processing" containerID="cri-o://cb05c3fffba49143b07146a728c341cfae9b5bd5ef7ef71b7e449b0abc59e443" gracePeriod=25 Nov 03 22:37:35.504378 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:35.504343 5084 generic.go:356] "Generic (PLEG): container finished" podID="ccd4b6d3-6913-4191-ac28-028701a386e8" containerID="0091922fcb609de295e6b47565f9e9d9a350642629a2c13ab4e06d8941c82dcf" exitCode=0 Nov 03 22:37:35.504676 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:35.504436 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4" event={"ID":"ccd4b6d3-6913-4191-ac28-028701a386e8","Type":"ContainerDied","Data":"0091922fcb609de295e6b47565f9e9d9a350642629a2c13ab4e06d8941c82dcf"} Nov 03 22:37:35.646612 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:35.645221 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4" Nov 03 22:37:35.724464 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:35.724422 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v" Nov 03 22:37:35.788990 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:35.788946 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae1546c-f272-4c56-8d56-5933d34c703b-kube-api-access-cx69k\" (UniqueName: \"kubernetes.io/projected/fae1546c-f272-4c56-8d56-5933d34c703b-kube-api-access-cx69k\") pod \"fae1546c-f272-4c56-8d56-5933d34c703b\" (UID: \"fae1546c-f272-4c56-8d56-5933d34c703b\") " Nov 03 22:37:35.789300 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:35.789225 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae1546c-f272-4c56-8d56-5933d34c703b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae1546c-f272-4c56-8d56-5933d34c703b-plugins-config-volume\") pod \"fae1546c-f272-4c56-8d56-5933d34c703b\" (UID: \"fae1546c-f272-4c56-8d56-5933d34c703b\") " Nov 03 22:37:35.789402 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:35.789384 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd4b6d3-6913-4191-ac28-028701a386e8-kube-api-access-vcpmq\" (UniqueName: \"kubernetes.io/projected/ccd4b6d3-6913-4191-ac28-028701a386e8-kube-api-access-vcpmq\") pod \"ccd4b6d3-6913-4191-ac28-028701a386e8\" (UID: \"ccd4b6d3-6913-4191-ac28-028701a386e8\") " Nov 03 22:37:35.789471 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:35.789428 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd4b6d3-6913-4191-ac28-028701a386e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd4b6d3-6913-4191-ac28-028701a386e8-plugins-config-volume\") pod \"ccd4b6d3-6913-4191-ac28-028701a386e8\" (UID: \"ccd4b6d3-6913-4191-ac28-028701a386e8\") " Nov 03 22:37:35.789658 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:35.789627 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae1546c-f272-4c56-8d56-5933d34c703b-plugins-config-volume" pod "fae1546c-f272-4c56-8d56-5933d34c703b" (UID: "fae1546c-f272-4c56-8d56-5933d34c703b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:35.790331 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:35.790273 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae1546c-f272-4c56-8d56-5933d34c703b-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:35.790530 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:35.790500 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd4b6d3-6913-4191-ac28-028701a386e8-plugins-config-volume" pod "ccd4b6d3-6913-4191-ac28-028701a386e8" (UID: "ccd4b6d3-6913-4191-ac28-028701a386e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:35.793161 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:35.793110 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd4b6d3-6913-4191-ac28-028701a386e8-kube-api-access-vcpmq" pod "ccd4b6d3-6913-4191-ac28-028701a386e8" (UID: "ccd4b6d3-6913-4191-ac28-028701a386e8"). InnerVolumeSpecName "kube-api-access-vcpmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:35.793419 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:35.793395 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae1546c-f272-4c56-8d56-5933d34c703b-kube-api-access-cx69k" pod "fae1546c-f272-4c56-8d56-5933d34c703b" (UID: "fae1546c-f272-4c56-8d56-5933d34c703b"). InnerVolumeSpecName "kube-api-access-cx69k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:35.891862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:35.891771 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx69k\" (UniqueName: \"kubernetes.io/projected/fae1546c-f272-4c56-8d56-5933d34c703b-kube-api-access-cx69k\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:35.891862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:35.891812 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcpmq\" (UniqueName: \"kubernetes.io/projected/ccd4b6d3-6913-4191-ac28-028701a386e8-kube-api-access-vcpmq\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:35.891862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:35.891829 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd4b6d3-6913-4191-ac28-028701a386e8-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:37:36.514264 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:36.514235 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4" Nov 03 22:37:36.514818 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:36.514278 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4" event={"ID":"ccd4b6d3-6913-4191-ac28-028701a386e8","Type":"ContainerDied","Data":"84e6dbfaee1c661dc173a10feb21521ed1bb4e6c12d013a9af32201990ccd6b5"} Nov 03 22:37:36.514818 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:36.514344 5084 scope.go:122] "RemoveContainer" containerID="0091922fcb609de295e6b47565f9e9d9a350642629a2c13ab4e06d8941c82dcf" Nov 03 22:37:36.516125 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:36.516095 5084 generic.go:356] "Generic (PLEG): container finished" podID="fae1546c-f272-4c56-8d56-5933d34c703b" containerID="cb05c3fffba49143b07146a728c341cfae9b5bd5ef7ef71b7e449b0abc59e443" exitCode=0 Nov 03 22:37:36.516263 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:36.516145 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v" Nov 03 22:37:36.516263 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:36.516139 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v" event={"ID":"fae1546c-f272-4c56-8d56-5933d34c703b","Type":"ContainerDied","Data":"cb05c3fffba49143b07146a728c341cfae9b5bd5ef7ef71b7e449b0abc59e443"} Nov 03 22:37:36.516263 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:36.516259 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v" event={"ID":"fae1546c-f272-4c56-8d56-5933d34c703b","Type":"ContainerDied","Data":"31fbaf2ed606aac9852c0cd018f459a77998370a0957f0aa76045414f0ed1947"} Nov 03 22:37:36.526555 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:36.526525 5084 scope.go:122] "RemoveContainer" containerID="cb05c3fffba49143b07146a728c341cfae9b5bd5ef7ef71b7e449b0abc59e443" Nov 03 22:37:36.537838 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:36.537816 5084 scope.go:122] "RemoveContainer" containerID="cb05c3fffba49143b07146a728c341cfae9b5bd5ef7ef71b7e449b0abc59e443" Nov 03 22:37:36.539877 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:37:36.539845 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb05c3fffba49143b07146a728c341cfae9b5bd5ef7ef71b7e449b0abc59e443\": container with ID starting with cb05c3fffba49143b07146a728c341cfae9b5bd5ef7ef71b7e449b0abc59e443 not found: ID does not exist" containerID="cb05c3fffba49143b07146a728c341cfae9b5bd5ef7ef71b7e449b0abc59e443" Nov 03 22:37:36.539991 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:36.539882 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb05c3fffba49143b07146a728c341cfae9b5bd5ef7ef71b7e449b0abc59e443"} err="failed to get container status \"cb05c3fffba49143b07146a728c341cfae9b5bd5ef7ef71b7e449b0abc59e443\": rpc error: code = NotFound desc = could not find container \"cb05c3fffba49143b07146a728c341cfae9b5bd5ef7ef71b7e449b0abc59e443\": container with ID starting with cb05c3fffba49143b07146a728c341cfae9b5bd5ef7ef71b7e449b0abc59e443 not found: ID does not exist" Nov 03 22:37:36.684342 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:36.684238 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v"] Nov 03 22:37:36.734493 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:36.734454 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-ngv6v"] Nov 03 22:37:36.854612 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:36.854560 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4"] Nov 03 22:37:36.866262 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:36.866212 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bc88eb47-7777b746f8-h4zs4"] Nov 03 22:37:38.630612 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:38.630574 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd4b6d3-6913-4191-ac28-028701a386e8" path="/var/lib/kubelet/pods/ccd4b6d3-6913-4191-ac28-028701a386e8/volumes" Nov 03 22:37:38.631116 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:38.631097 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae1546c-f272-4c56-8d56-5933d34c703b" path="/var/lib/kubelet/pods/fae1546c-f272-4c56-8d56-5933d34c703b/volumes" Nov 03 22:37:42.098246 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:37:42.098211 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:37:53.265117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.260488 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t"] Nov 03 22:37:53.266977 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.266935 5084 cpu_manager.go:414] "containerMap: removing container" podUID="ccd4b6d3-6913-4191-ac28-028701a386e8" containerName="payload-processing" Nov 03 22:37:53.267224 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.267200 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd4b6d3-6913-4191-ac28-028701a386e8" containerName="payload-processing" Nov 03 22:37:53.267350 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.267339 5084 cpu_manager.go:414] "containerMap: removing container" podUID="5be671e7-45d4-4e28-b384-2f3dd6397ecb" containerName="istio-proxy" Nov 03 22:37:53.267446 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.267435 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be671e7-45d4-4e28-b384-2f3dd6397ecb" containerName="istio-proxy" Nov 03 22:37:53.267545 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.267534 5084 cpu_manager.go:414] "containerMap: removing container" podUID="fae1546c-f272-4c56-8d56-5933d34c703b" containerName="payload-pre-processing" Nov 03 22:37:53.267623 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.267609 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae1546c-f272-4c56-8d56-5933d34c703b" containerName="payload-pre-processing" Nov 03 22:37:53.267958 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.267936 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd4b6d3-6913-4191-ac28-028701a386e8" containerName="payload-processing" Nov 03 22:37:53.268102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.268086 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="5be671e7-45d4-4e28-b384-2f3dd6397ecb" containerName="istio-proxy" Nov 03 22:37:53.268229 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.268216 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae1546c-f272-4c56-8d56-5933d34c703b" containerName="payload-pre-processing" Nov 03 22:37:53.294310 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.294273 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:37:53.316079 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.316022 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t"] Nov 03 22:37:53.369089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.366368 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-fac3b6-serving-cert\"" Nov 03 22:37:53.406226 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.406191 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef54935-c879-44c3-b864-fff0bbfbcfd9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t\" (UID: \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:37:53.406643 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.406620 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ef54935-c879-44c3-b864-fff0bbfbcfd9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t\" (UID: \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:37:53.406857 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.406839 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54rcc\" (UniqueName: \"kubernetes.io/projected/1ef54935-c879-44c3-b864-fff0bbfbcfd9-kube-api-access-54rcc\") pod \"maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t\" (UID: \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:37:53.407120 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.407095 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ef54935-c879-44c3-b864-fff0bbfbcfd9-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t\" (UID: \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:37:53.530153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.528974 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ef54935-c879-44c3-b864-fff0bbfbcfd9-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t\" (UID: \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:37:53.530153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.529205 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef54935-c879-44c3-b864-fff0bbfbcfd9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t\" (UID: \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:37:53.530153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.529350 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ef54935-c879-44c3-b864-fff0bbfbcfd9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t\" (UID: \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:37:53.530153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.529387 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54rcc\" (UniqueName: \"kubernetes.io/projected/1ef54935-c879-44c3-b864-fff0bbfbcfd9-kube-api-access-54rcc\") pod \"maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t\" (UID: \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:37:53.568093 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:37:53.556012 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-serving-cert: secret "maas-api-e2e-shared-a-w5-fac3b6-serving-cert" not found Nov 03 22:37:53.568470 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:37:53.568445 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ef54935-c879-44c3-b864-fff0bbfbcfd9-maas-api-tls podName:1ef54935-c879-44c3-b864-fff0bbfbcfd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:37:54.068412571 +0000 UTC m=+1841.662444541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ef54935-c879-44c3-b864-fff0bbfbcfd9-maas-api-tls") pod "maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" (UID: "1ef54935-c879-44c3-b864-fff0bbfbcfd9") : secret "maas-api-e2e-shared-a-w5-fac3b6-serving-cert" not found Nov 03 22:37:53.579186 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.563620 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef54935-c879-44c3-b864-fff0bbfbcfd9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t\" (UID: \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:37:53.587098 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.582305 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ef54935-c879-44c3-b864-fff0bbfbcfd9-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t\" (UID: \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:37:53.619200 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:53.619164 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54rcc\" (UniqueName: \"kubernetes.io/projected/1ef54935-c879-44c3-b864-fff0bbfbcfd9-kube-api-access-54rcc\") pod \"maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t\" (UID: \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:37:54.150715 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:54.150675 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ef54935-c879-44c3-b864-fff0bbfbcfd9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t\" (UID: \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:37:54.153205 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:54.153173 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ef54935-c879-44c3-b864-fff0bbfbcfd9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t\" (UID: \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:37:54.211292 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:54.211257 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:37:54.637278 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:54.635165 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t"] Nov 03 22:37:54.895692 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:54.895591 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" event={"ID":"1ef54935-c879-44c3-b864-fff0bbfbcfd9","Type":"ContainerStarted","Data":"c73016da52ae42d7f885a22441c6bcd3f931980fab7d5fe8249826eff0ac1c5c"} Nov 03 22:37:55.906354 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:55.906262 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" event={"ID":"1ef54935-c879-44c3-b864-fff0bbfbcfd9","Type":"ContainerStarted","Data":"b8b308609580eefd97e5529b2033b7c55d29f183b1db780ae7ddd362f0f29158"} Nov 03 22:37:55.906808 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:55.906781 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:37:56.032591 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:37:56.032516 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" podStartSLOduration=2.071905701 podStartE2EDuration="3.032494341s" totalImagesPullingTime="960.58864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:56.027265011 +0000 UTC m=+1843.621296985" watchObservedRunningTime="2026-11-03 22:37:56.032494341 +0000 UTC m=+1843.626526316" Nov 03 22:38:01.920847 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:01.920811 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:38:05.450545 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:05.450505 5084 generic.go:356] "Generic (PLEG): container finished" podID="507503d1-913a-4eba-8145-4818fc379376" containerID="3da55bf7ce31e3848f6d4c61d2ff1fd075642f74d63ad136b9266400a53b7c7d" exitCode=0 Nov 03 22:38:05.450985 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:05.450576 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" event={"ID":"507503d1-913a-4eba-8145-4818fc379376","Type":"ContainerDied","Data":"3da55bf7ce31e3848f6d4c61d2ff1fd075642f74d63ad136b9266400a53b7c7d"} Nov 03 22:38:05.451171 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:05.451150 5084 scope.go:122] "RemoveContainer" containerID="3da55bf7ce31e3848f6d4c61d2ff1fd075642f74d63ad136b9266400a53b7c7d" Nov 03 22:38:05.453284 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:05.453238 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:38:05.453406 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:05.453336 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:38:05.453406 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:05.453370 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:38:06.467497 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:06.466844 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" event={"ID":"507503d1-913a-4eba-8145-4818fc379376","Type":"ContainerStarted","Data":"0bf3c9431457a8794f050d744a39c4c03e4c4ea610ec5cfce3632b7c776e355c"} Nov 03 22:38:07.399162 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:07.399117 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:38:07.400498 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:07.400467 5084 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 22:38:07.400655 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:07.400532 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" podUID="507503d1-913a-4eba-8145-4818fc379376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 22:38:08.399899 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:08.399852 5084 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 22:38:08.400481 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:08.399914 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" podUID="507503d1-913a-4eba-8145-4818fc379376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 22:38:09.046437 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.046401 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph"] Nov 03 22:38:09.051572 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.051537 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:38:09.105503 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.105462 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph"] Nov 03 22:38:09.151724 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.151651 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-a92c9e-serving-cert\"" Nov 03 22:38:09.188876 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.188835 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph\" (UID: \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:38:09.189169 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.188943 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bztf\" (UniqueName: \"kubernetes.io/projected/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-kube-api-access-9bztf\") pod \"maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph\" (UID: \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:38:09.189305 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.189184 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph\" (UID: \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:38:09.189413 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.189329 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph\" (UID: \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:38:09.278102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.276173 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:38:09.291227 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.291186 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph\" (UID: \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:38:09.291427 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.291245 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bztf\" (UniqueName: \"kubernetes.io/projected/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-kube-api-access-9bztf\") pod \"maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph\" (UID: \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:38:09.291427 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.291416 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph\" (UID: \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:38:09.291517 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.291505 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph\" (UID: \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:38:09.291771 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:38:09.291751 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-serving-cert: secret "maas-api-e2e-shared-b-w4-a92c9e-serving-cert" not found Nov 03 22:38:09.291870 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:38:09.291820 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-maas-api-tls podName:ba7426c0-eb6f-48f6-bc42-3232fefe97a4 nodeName:}" failed. No retries permitted until 2026-11-03 22:38:09.79179865 +0000 UTC m=+1857.385830616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-maas-api-tls") pod "maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" (UID: "ba7426c0-eb6f-48f6-bc42-3232fefe97a4") : secret "maas-api-e2e-shared-b-w4-a92c9e-serving-cert" not found Nov 03 22:38:09.292649 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.292621 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph\" (UID: \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:38:09.294558 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.294529 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph\" (UID: \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:38:09.411148 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.396509 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bztf\" (UniqueName: \"kubernetes.io/projected/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-kube-api-access-9bztf\") pod \"maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph\" (UID: \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:38:09.411148 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.409220 5084 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 22:38:09.411148 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.409277 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" podUID="507503d1-913a-4eba-8145-4818fc379376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 22:38:09.822962 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.822908 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph\" (UID: \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:38:09.826446 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.826408 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph\" (UID: \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:38:09.990486 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:09.990448 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:38:10.507850 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:10.507213 5084 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 22:38:10.507850 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:10.507274 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" podUID="507503d1-913a-4eba-8145-4818fc379376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 22:38:10.526703 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:10.526662 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph"] Nov 03 22:38:10.655130 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:38:10.651445 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7426c0_eb6f_48f6_bc42_3232fefe97a4.slice/crio-6108df4cfd9346050e8f5934ad3c54f9ba8a619f14ea70569132b1b8a46cfdc0 WatchSource:0}: Error finding container 6108df4cfd9346050e8f5934ad3c54f9ba8a619f14ea70569132b1b8a46cfdc0: Status 404 returned error can't find the container with id 6108df4cfd9346050e8f5934ad3c54f9ba8a619f14ea70569132b1b8a46cfdc0 Nov 03 22:38:11.400083 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:11.399892 5084 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 22:38:11.400083 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:11.399959 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" podUID="507503d1-913a-4eba-8145-4818fc379376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 22:38:11.582011 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:11.581926 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" event={"ID":"ba7426c0-eb6f-48f6-bc42-3232fefe97a4","Type":"ContainerStarted","Data":"6108df4cfd9346050e8f5934ad3c54f9ba8a619f14ea70569132b1b8a46cfdc0"} Nov 03 22:38:12.031212 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:38:12.031173 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:38:12.511192 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:12.511155 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:38:12.512688 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:12.512648 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:38:12.592413 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:12.592378 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" event={"ID":"ba7426c0-eb6f-48f6-bc42-3232fefe97a4","Type":"ContainerStarted","Data":"dd16872db338aa60f053d2a3b172685f4f55260e8dbf71412ceb4a5b6a34ae80"} Nov 03 22:38:12.592943 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:12.592467 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:38:12.793404 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:12.793333 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" podStartSLOduration=3.626853096 podStartE2EDuration="4.793309372s" totalImagesPullingTime="1.166456276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:12.790334135 +0000 UTC m=+1860.384366112" watchObservedRunningTime="2026-11-03 22:38:12.793309372 +0000 UTC m=+1860.387341348" Nov 03 22:38:18.942781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:18.942750 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:38:27.474587 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:27.474538 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8"] Nov 03 22:38:27.475170 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:27.474856 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" podUID="50e61dd6-336d-48cb-bda1-7ee80c1e27b9" containerName="istio-proxy" containerID="cri-o://108f946da24fcfe10d6548ae236e36d49220385fc76ee76ea6626c6ccd567f66" gracePeriod=30 Nov 03 22:38:33.076964 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.076924 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:38:33.144542 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.144508 5084 generic.go:356] "Generic (PLEG): container finished" podID="50e61dd6-336d-48cb-bda1-7ee80c1e27b9" containerID="108f946da24fcfe10d6548ae236e36d49220385fc76ee76ea6626c6ccd567f66" exitCode=0 Nov 03 22:38:33.144750 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.144558 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" Nov 03 22:38:33.144750 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.144595 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" event={"ID":"50e61dd6-336d-48cb-bda1-7ee80c1e27b9","Type":"ContainerDied","Data":"108f946da24fcfe10d6548ae236e36d49220385fc76ee76ea6626c6ccd567f66"} Nov 03 22:38:33.144750 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.144652 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8" event={"ID":"50e61dd6-336d-48cb-bda1-7ee80c1e27b9","Type":"ContainerDied","Data":"bdd809fcff41d11731fcd2acb315a6296ae9eb122647bcbe43767bc229f685e6"} Nov 03 22:38:33.144750 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.144674 5084 scope.go:122] "RemoveContainer" containerID="108f946da24fcfe10d6548ae236e36d49220385fc76ee76ea6626c6ccd567f66" Nov 03 22:38:33.156264 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.156237 5084 scope.go:122] "RemoveContainer" containerID="108f946da24fcfe10d6548ae236e36d49220385fc76ee76ea6626c6ccd567f66" Nov 03 22:38:33.156634 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:38:33.156605 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"108f946da24fcfe10d6548ae236e36d49220385fc76ee76ea6626c6ccd567f66\": container with ID starting with 108f946da24fcfe10d6548ae236e36d49220385fc76ee76ea6626c6ccd567f66 not found: ID does not exist" containerID="108f946da24fcfe10d6548ae236e36d49220385fc76ee76ea6626c6ccd567f66" Nov 03 22:38:33.156719 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.156645 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"108f946da24fcfe10d6548ae236e36d49220385fc76ee76ea6626c6ccd567f66"} err="failed to get container status \"108f946da24fcfe10d6548ae236e36d49220385fc76ee76ea6626c6ccd567f66\": rpc error: code = NotFound desc = could not find container \"108f946da24fcfe10d6548ae236e36d49220385fc76ee76ea6626c6ccd567f66\": container with ID starting with 108f946da24fcfe10d6548ae236e36d49220385fc76ee76ea6626c6ccd567f66 not found: ID does not exist" Nov 03 22:38:33.185466 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.185426 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-workload-socket\") pod \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " Nov 03 22:38:33.185668 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.185488 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-token\" (UniqueName: \"kubernetes.io/projected/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-token\") pod \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " Nov 03 22:38:33.185668 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.185575 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-credential-socket\") pod \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " Nov 03 22:38:33.185668 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.185640 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-kube-api-access-c2qss\" (UniqueName: \"kubernetes.io/projected/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-kube-api-access-c2qss\") pod \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " Nov 03 22:38:33.185923 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.185717 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-workload-certs\") pod \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " Nov 03 22:38:33.185923 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.185789 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-envoy\") pod \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " Nov 03 22:38:33.185923 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.185815 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-podinfo\") pod \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " Nov 03 22:38:33.185923 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.185885 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-data\") pod \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " Nov 03 22:38:33.186196 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.185998 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-workload-socket" pod "50e61dd6-336d-48cb-bda1-7ee80c1e27b9" (UID: "50e61dd6-336d-48cb-bda1-7ee80c1e27b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:33.186196 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.186031 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istiod-ca-cert\") pod \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\" (UID: \"50e61dd6-336d-48cb-bda1-7ee80c1e27b9\") " Nov 03 22:38:33.186196 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.186178 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-workload-certs" pod "50e61dd6-336d-48cb-bda1-7ee80c1e27b9" (UID: "50e61dd6-336d-48cb-bda1-7ee80c1e27b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:33.186538 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.186506 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-data" pod "50e61dd6-336d-48cb-bda1-7ee80c1e27b9" (UID: "50e61dd6-336d-48cb-bda1-7ee80c1e27b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:33.186910 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.186885 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istiod-ca-cert" pod "50e61dd6-336d-48cb-bda1-7ee80c1e27b9" (UID: "50e61dd6-336d-48cb-bda1-7ee80c1e27b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:33.187049 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.186976 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:33.187049 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.187002 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:33.187049 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.187018 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:33.201430 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.188969 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-podinfo" pod "50e61dd6-336d-48cb-bda1-7ee80c1e27b9" (UID: "50e61dd6-336d-48cb-bda1-7ee80c1e27b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:38:33.201430 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.189031 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-token" pod "50e61dd6-336d-48cb-bda1-7ee80c1e27b9" (UID: "50e61dd6-336d-48cb-bda1-7ee80c1e27b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:33.201430 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.192697 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-credential-socket" pod "50e61dd6-336d-48cb-bda1-7ee80c1e27b9" (UID: "50e61dd6-336d-48cb-bda1-7ee80c1e27b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:33.201430 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.193551 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-kube-api-access-c2qss" pod "50e61dd6-336d-48cb-bda1-7ee80c1e27b9" (UID: "50e61dd6-336d-48cb-bda1-7ee80c1e27b9"). InnerVolumeSpecName "kube-api-access-c2qss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:33.201430 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.196241 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-envoy" pod "50e61dd6-336d-48cb-bda1-7ee80c1e27b9" (UID: "50e61dd6-336d-48cb-bda1-7ee80c1e27b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:33.288311 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.288268 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:33.288311 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.288308 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:33.288552 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.288322 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:33.288552 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.288335 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:33.288552 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.288347 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:33.288552 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.288359 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2qss\" (UniqueName: \"kubernetes.io/projected/50e61dd6-336d-48cb-bda1-7ee80c1e27b9-kube-api-access-c2qss\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:33.678173 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.675684 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8"] Nov 03 22:38:33.684492 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:33.684452 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-d99h8"] Nov 03 22:38:34.631036 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:34.630987 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50e61dd6-336d-48cb-bda1-7ee80c1e27b9" path="/var/lib/kubelet/pods/50e61dd6-336d-48cb-bda1-7ee80c1e27b9/volumes" Nov 03 22:38:41.482521 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:41.482479 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk"] Nov 03 22:38:41.483151 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:41.482766 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerName="istio-proxy" containerID="cri-o://2f6c876dfb10ab128f9ce00f36b56008a8317a0fb6582897c2d2c982dbd48a8b" gracePeriod=30 Nov 03 22:38:41.710158 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:41.710109 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6"] Nov 03 22:38:41.710443 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:41.710400 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" podUID="6ca4f5ec-9244-447b-ba40-91409196e307" containerName="istio-proxy" containerID="cri-o://976a58303b267c6211f69c2aa52cb44bfcc6c1b88751948cffedbabdcf4a64df" gracePeriod=30 Nov 03 22:38:41.718813 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:41.716958 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982"] Nov 03 22:38:41.718813 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:41.717236 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" podUID="93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" containerName="istio-proxy" containerID="cri-o://3d8fafceef96b31382c67eeb1086afd63d77914c90f6710de3d67a4774cfdadd" gracePeriod=30 Nov 03 22:38:42.025782 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:38:42.025742 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:38:42.117551 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:42.117509 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp"] Nov 03 22:38:42.117774 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:42.117742 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" podUID="487f04a7-abb9-40e2-9744-7068195f5e00" containerName="istio-proxy" containerID="cri-o://19c33cfca9ed5d86c52f78b598c2ac06301559c652e23397ae53c5ef8e327ba1" gracePeriod=30 Nov 03 22:38:43.551123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:43.551043 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:38:43.551773 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:43.551738 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" podUID="487f04a7-abb9-40e2-9744-7068195f5e00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:38:46.953154 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:46.952682 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:38:46.997536 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:46.993153 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:38:47.081671 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.081628 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-token\" (UniqueName: \"kubernetes.io/projected/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-token\") pod \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " Nov 03 22:38:47.081907 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.081725 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc48515-bea2-4a71-8d48-c6000b2a46d9-kube-api-access-q4l4x\" (UniqueName: \"kubernetes.io/projected/cfc48515-bea2-4a71-8d48-c6000b2a46d9-kube-api-access-q4l4x\") pod \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " Nov 03 22:38:47.081982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.081887 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ca4f5ec-9244-447b-ba40-91409196e307-kube-api-access-qt5nt\" (UniqueName: \"kubernetes.io/projected/6ca4f5ec-9244-447b-ba40-91409196e307-kube-api-access-qt5nt\") pod \"6ca4f5ec-9244-447b-ba40-91409196e307\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " Nov 03 22:38:47.081982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.081956 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-envoy\") pod \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " Nov 03 22:38:47.082108 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.082088 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-istio-data\") pod \"6ca4f5ec-9244-447b-ba40-91409196e307\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " Nov 03 22:38:47.082168 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.082119 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-credential-socket\") pod \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " Nov 03 22:38:47.082224 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.082189 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-credential-socket\") pod \"6ca4f5ec-9244-447b-ba40-91409196e307\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " Nov 03 22:38:47.082283 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.082222 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istiod-ca-cert\") pod \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " Nov 03 22:38:47.082283 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.082250 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-podinfo\") pod \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " Nov 03 22:38:47.082423 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.082282 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-istio-envoy\") pod \"6ca4f5ec-9244-447b-ba40-91409196e307\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " Nov 03 22:38:47.082423 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.082337 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-workload-certs\") pod \"6ca4f5ec-9244-447b-ba40-91409196e307\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " Nov 03 22:38:47.082423 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.082393 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-workload-socket\") pod \"6ca4f5ec-9244-447b-ba40-91409196e307\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " Nov 03 22:38:47.082591 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.082428 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-data\") pod \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " Nov 03 22:38:47.082591 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.082489 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ca4f5ec-9244-447b-ba40-91409196e307-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca4f5ec-9244-447b-ba40-91409196e307-istiod-ca-cert\") pod \"6ca4f5ec-9244-447b-ba40-91409196e307\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " Nov 03 22:38:47.082591 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.082517 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-workload-socket\") pod \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " Nov 03 22:38:47.083046 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.082605 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ca4f5ec-9244-447b-ba40-91409196e307-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca4f5ec-9244-447b-ba40-91409196e307-istio-podinfo\") pod \"6ca4f5ec-9244-447b-ba40-91409196e307\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " Nov 03 22:38:47.083046 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.082653 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-workload-certs\") pod \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\" (UID: \"cfc48515-bea2-4a71-8d48-c6000b2a46d9\") " Nov 03 22:38:47.083046 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.082692 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ca4f5ec-9244-447b-ba40-91409196e307-istio-token\" (UniqueName: \"kubernetes.io/projected/6ca4f5ec-9244-447b-ba40-91409196e307-istio-token\") pod \"6ca4f5ec-9244-447b-ba40-91409196e307\" (UID: \"6ca4f5ec-9244-447b-ba40-91409196e307\") " Nov 03 22:38:47.085489 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.085265 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-workload-socket" pod "cfc48515-bea2-4a71-8d48-c6000b2a46d9" (UID: "cfc48515-bea2-4a71-8d48-c6000b2a46d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.086297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.085714 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-workload-certs" pod "6ca4f5ec-9244-447b-ba40-91409196e307" (UID: "6ca4f5ec-9244-447b-ba40-91409196e307"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.086297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.086076 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-data" pod "cfc48515-bea2-4a71-8d48-c6000b2a46d9" (UID: "cfc48515-bea2-4a71-8d48-c6000b2a46d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.087113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.086651 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca4f5ec-9244-447b-ba40-91409196e307-istiod-ca-cert" pod "6ca4f5ec-9244-447b-ba40-91409196e307" (UID: "6ca4f5ec-9244-447b-ba40-91409196e307"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:47.087113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.086927 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-workload-socket" pod "6ca4f5ec-9244-447b-ba40-91409196e307" (UID: "6ca4f5ec-9244-447b-ba40-91409196e307"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.087662 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.087333 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-credential-socket" pod "6ca4f5ec-9244-447b-ba40-91409196e307" (UID: "6ca4f5ec-9244-447b-ba40-91409196e307"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.088041 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.087817 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-credential-socket" pod "cfc48515-bea2-4a71-8d48-c6000b2a46d9" (UID: "cfc48515-bea2-4a71-8d48-c6000b2a46d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.088222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.088192 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-istio-data" pod "6ca4f5ec-9244-447b-ba40-91409196e307" (UID: "6ca4f5ec-9244-447b-ba40-91409196e307"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.089422 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.089382 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-workload-certs" pod "cfc48515-bea2-4a71-8d48-c6000b2a46d9" (UID: "cfc48515-bea2-4a71-8d48-c6000b2a46d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.089589 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.089523 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istiod-ca-cert" pod "cfc48515-bea2-4a71-8d48-c6000b2a46d9" (UID: "cfc48515-bea2-4a71-8d48-c6000b2a46d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:47.089894 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.089867 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-envoy" pod "cfc48515-bea2-4a71-8d48-c6000b2a46d9" (UID: "cfc48515-bea2-4a71-8d48-c6000b2a46d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.090209 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.090181 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc48515-bea2-4a71-8d48-c6000b2a46d9-kube-api-access-q4l4x" pod "cfc48515-bea2-4a71-8d48-c6000b2a46d9" (UID: "cfc48515-bea2-4a71-8d48-c6000b2a46d9"). InnerVolumeSpecName "kube-api-access-q4l4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:47.090641 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.090606 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca4f5ec-9244-447b-ba40-91409196e307-istio-token" pod "6ca4f5ec-9244-447b-ba40-91409196e307" (UID: "6ca4f5ec-9244-447b-ba40-91409196e307"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:47.091505 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.091468 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-token" pod "cfc48515-bea2-4a71-8d48-c6000b2a46d9" (UID: "cfc48515-bea2-4a71-8d48-c6000b2a46d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:47.091674 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.091646 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-podinfo" pod "cfc48515-bea2-4a71-8d48-c6000b2a46d9" (UID: "cfc48515-bea2-4a71-8d48-c6000b2a46d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:38:47.092175 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.092138 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca4f5ec-9244-447b-ba40-91409196e307-kube-api-access-qt5nt" pod "6ca4f5ec-9244-447b-ba40-91409196e307" (UID: "6ca4f5ec-9244-447b-ba40-91409196e307"). InnerVolumeSpecName "kube-api-access-qt5nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:47.092482 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.092456 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ca4f5ec-9244-447b-ba40-91409196e307-istio-podinfo" pod "6ca4f5ec-9244-447b-ba40-91409196e307" (UID: "6ca4f5ec-9244-447b-ba40-91409196e307"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:38:47.092780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.092728 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-istio-envoy" pod "6ca4f5ec-9244-447b-ba40-91409196e307" (UID: "6ca4f5ec-9244-447b-ba40-91409196e307"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.128934 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.122739 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:38:47.187258 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.187218 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.187258 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.187260 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.187639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.187274 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.187639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.187290 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.187639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.187304 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.187639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.187319 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.187639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.187332 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.187639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.187346 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca4f5ec-9244-447b-ba40-91409196e307-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.187639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.187360 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.187639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.187373 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca4f5ec-9244-447b-ba40-91409196e307-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.187639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.187385 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.187639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.187399 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca4f5ec-9244-447b-ba40-91409196e307-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.187639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.187413 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.187639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.187426 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca4f5ec-9244-447b-ba40-91409196e307-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.187639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.187440 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.187639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.187455 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4l4x\" (UniqueName: \"kubernetes.io/projected/cfc48515-bea2-4a71-8d48-c6000b2a46d9-kube-api-access-q4l4x\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.187639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.187483 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt5nt\" (UniqueName: \"kubernetes.io/projected/6ca4f5ec-9244-447b-ba40-91409196e307-kube-api-access-qt5nt\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.187639 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.187499 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfc48515-bea2-4a71-8d48-c6000b2a46d9-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.288756 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.288427 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-workload-certs\") pod \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " Nov 03 22:38:47.288756 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.288621 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-podinfo\") pod \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " Nov 03 22:38:47.288756 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.288685 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-data\") pod \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " Nov 03 22:38:47.288756 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.288735 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-workload-socket\") pod \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " Nov 03 22:38:47.289189 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.288786 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-envoy\") pod \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " Nov 03 22:38:47.289189 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.288835 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istiod-ca-cert\") pod \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " Nov 03 22:38:47.289189 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.288866 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-workload-certs" pod "93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" (UID: "93bd9f01-32b4-42ae-9ef8-f0bef05b18b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.289189 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.288913 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-token\" (UniqueName: \"kubernetes.io/projected/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-token\") pod \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " Nov 03 22:38:47.289189 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.288990 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-kube-api-access-2nd4j\" (UniqueName: \"kubernetes.io/projected/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-kube-api-access-2nd4j\") pod \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " Nov 03 22:38:47.289189 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.289018 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-credential-socket\") pod \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\" (UID: \"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2\") " Nov 03 22:38:47.289189 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.289113 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-data" pod "93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" (UID: "93bd9f01-32b4-42ae-9ef8-f0bef05b18b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.289800 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.289772 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.289911 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.289804 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.290591 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.290557 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-credential-socket" pod "93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" (UID: "93bd9f01-32b4-42ae-9ef8-f0bef05b18b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.290800 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.290779 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-workload-socket" pod "93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" (UID: "93bd9f01-32b4-42ae-9ef8-f0bef05b18b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.291986 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.291953 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-podinfo" pod "93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" (UID: "93bd9f01-32b4-42ae-9ef8-f0bef05b18b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:38:47.292417 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.292391 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istiod-ca-cert" pod "93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" (UID: "93bd9f01-32b4-42ae-9ef8-f0bef05b18b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:47.297653 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.294249 5084 generic.go:356] "Generic (PLEG): container finished" podID="93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" containerID="3d8fafceef96b31382c67eeb1086afd63d77914c90f6710de3d67a4774cfdadd" exitCode=0 Nov 03 22:38:47.297653 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.294347 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" event={"ID":"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2","Type":"ContainerDied","Data":"3d8fafceef96b31382c67eeb1086afd63d77914c90f6710de3d67a4774cfdadd"} Nov 03 22:38:47.297653 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.294401 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" event={"ID":"93bd9f01-32b4-42ae-9ef8-f0bef05b18b2","Type":"ContainerDied","Data":"7244b61f63ecb2da7c7c06dda52de420fdacc8711a91758de2e2f21d02d83fd3"} Nov 03 22:38:47.297653 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.294571 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-kube-api-access-2nd4j" pod "93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" (UID: "93bd9f01-32b4-42ae-9ef8-f0bef05b18b2"). InnerVolumeSpecName "kube-api-access-2nd4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:47.297653 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.294732 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982" Nov 03 22:38:47.297653 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.294990 5084 scope.go:122] "RemoveContainer" containerID="3d8fafceef96b31382c67eeb1086afd63d77914c90f6710de3d67a4774cfdadd" Nov 03 22:38:47.306889 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.299401 5084 generic.go:356] "Generic (PLEG): container finished" podID="487f04a7-abb9-40e2-9744-7068195f5e00" containerID="19c33cfca9ed5d86c52f78b598c2ac06301559c652e23397ae53c5ef8e327ba1" exitCode=0 Nov 03 22:38:47.306889 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.299619 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" event={"ID":"487f04a7-abb9-40e2-9744-7068195f5e00","Type":"ContainerDied","Data":"19c33cfca9ed5d86c52f78b598c2ac06301559c652e23397ae53c5ef8e327ba1"} Nov 03 22:38:47.306889 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.303015 5084 generic.go:356] "Generic (PLEG): container finished" podID="6ca4f5ec-9244-447b-ba40-91409196e307" containerID="976a58303b267c6211f69c2aa52cb44bfcc6c1b88751948cffedbabdcf4a64df" exitCode=0 Nov 03 22:38:47.306889 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.303248 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" event={"ID":"6ca4f5ec-9244-447b-ba40-91409196e307","Type":"ContainerDied","Data":"976a58303b267c6211f69c2aa52cb44bfcc6c1b88751948cffedbabdcf4a64df"} Nov 03 22:38:47.306889 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.303302 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" event={"ID":"6ca4f5ec-9244-447b-ba40-91409196e307","Type":"ContainerDied","Data":"6fb1bc00dbb910478b07f6118d87cae687716636b5dbd7e8a8f45ec7458335fd"} Nov 03 22:38:47.306889 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.303396 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6" Nov 03 22:38:47.306889 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.304423 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-envoy" pod "93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" (UID: "93bd9f01-32b4-42ae-9ef8-f0bef05b18b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.310278 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.308396 5084 generic.go:356] "Generic (PLEG): container finished" podID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerID="2f6c876dfb10ab128f9ce00f36b56008a8317a0fb6582897c2d2c982dbd48a8b" exitCode=0 Nov 03 22:38:47.310278 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.308459 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" event={"ID":"cfc48515-bea2-4a71-8d48-c6000b2a46d9","Type":"ContainerDied","Data":"2f6c876dfb10ab128f9ce00f36b56008a8317a0fb6582897c2d2c982dbd48a8b"} Nov 03 22:38:47.310278 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.308510 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" event={"ID":"cfc48515-bea2-4a71-8d48-c6000b2a46d9","Type":"ContainerDied","Data":"5f6e3aa8aa4ddf736bc88ac1c72f3c1b8db45cc3b186f5a96beb05c2fc508600"} Nov 03 22:38:47.310278 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.308592 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk" Nov 03 22:38:47.310278 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.310259 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-token" pod "93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" (UID: "93bd9f01-32b4-42ae-9ef8-f0bef05b18b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:47.318689 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.318658 5084 scope.go:122] "RemoveContainer" containerID="3d8fafceef96b31382c67eeb1086afd63d77914c90f6710de3d67a4774cfdadd" Nov 03 22:38:47.319168 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:38:47.319053 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d8fafceef96b31382c67eeb1086afd63d77914c90f6710de3d67a4774cfdadd\": container with ID starting with 3d8fafceef96b31382c67eeb1086afd63d77914c90f6710de3d67a4774cfdadd not found: ID does not exist" containerID="3d8fafceef96b31382c67eeb1086afd63d77914c90f6710de3d67a4774cfdadd" Nov 03 22:38:47.319290 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.319180 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d8fafceef96b31382c67eeb1086afd63d77914c90f6710de3d67a4774cfdadd"} err="failed to get container status \"3d8fafceef96b31382c67eeb1086afd63d77914c90f6710de3d67a4774cfdadd\": rpc error: code = NotFound desc = could not find container \"3d8fafceef96b31382c67eeb1086afd63d77914c90f6710de3d67a4774cfdadd\": container with ID starting with 3d8fafceef96b31382c67eeb1086afd63d77914c90f6710de3d67a4774cfdadd not found: ID does not exist" Nov 03 22:38:47.319290 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.319205 5084 scope.go:122] "RemoveContainer" containerID="976a58303b267c6211f69c2aa52cb44bfcc6c1b88751948cffedbabdcf4a64df" Nov 03 22:38:47.332610 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.332172 5084 scope.go:122] "RemoveContainer" containerID="976a58303b267c6211f69c2aa52cb44bfcc6c1b88751948cffedbabdcf4a64df" Nov 03 22:38:47.332610 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:38:47.332592 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"976a58303b267c6211f69c2aa52cb44bfcc6c1b88751948cffedbabdcf4a64df\": container with ID starting with 976a58303b267c6211f69c2aa52cb44bfcc6c1b88751948cffedbabdcf4a64df not found: ID does not exist" containerID="976a58303b267c6211f69c2aa52cb44bfcc6c1b88751948cffedbabdcf4a64df" Nov 03 22:38:47.332772 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.332632 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976a58303b267c6211f69c2aa52cb44bfcc6c1b88751948cffedbabdcf4a64df"} err="failed to get container status \"976a58303b267c6211f69c2aa52cb44bfcc6c1b88751948cffedbabdcf4a64df\": rpc error: code = NotFound desc = could not find container \"976a58303b267c6211f69c2aa52cb44bfcc6c1b88751948cffedbabdcf4a64df\": container with ID starting with 976a58303b267c6211f69c2aa52cb44bfcc6c1b88751948cffedbabdcf4a64df not found: ID does not exist" Nov 03 22:38:47.332772 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.332659 5084 scope.go:122] "RemoveContainer" containerID="2f6c876dfb10ab128f9ce00f36b56008a8317a0fb6582897c2d2c982dbd48a8b" Nov 03 22:38:47.382162 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.369180 5084 scope.go:122] "RemoveContainer" containerID="2f6c876dfb10ab128f9ce00f36b56008a8317a0fb6582897c2d2c982dbd48a8b" Nov 03 22:38:47.390405 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:38:47.390355 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f6c876dfb10ab128f9ce00f36b56008a8317a0fb6582897c2d2c982dbd48a8b\": container with ID starting with 2f6c876dfb10ab128f9ce00f36b56008a8317a0fb6582897c2d2c982dbd48a8b not found: ID does not exist" containerID="2f6c876dfb10ab128f9ce00f36b56008a8317a0fb6582897c2d2c982dbd48a8b" Nov 03 22:38:47.390569 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.390404 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6c876dfb10ab128f9ce00f36b56008a8317a0fb6582897c2d2c982dbd48a8b"} err="failed to get container status \"2f6c876dfb10ab128f9ce00f36b56008a8317a0fb6582897c2d2c982dbd48a8b\": rpc error: code = NotFound desc = could not find container \"2f6c876dfb10ab128f9ce00f36b56008a8317a0fb6582897c2d2c982dbd48a8b\": container with ID starting with 2f6c876dfb10ab128f9ce00f36b56008a8317a0fb6582897c2d2c982dbd48a8b not found: ID does not exist" Nov 03 22:38:47.392109 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.391480 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.392109 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.391518 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.392109 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.391532 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.392109 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.391548 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.392109 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.391562 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.392109 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.391576 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nd4j\" (UniqueName: \"kubernetes.io/projected/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-kube-api-access-2nd4j\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.392109 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.391590 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.461491 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.461455 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:38:47.513195 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.513158 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6"] Nov 03 22:38:47.518200 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.518042 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-98tt6"] Nov 03 22:38:47.593678 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.593569 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/487f04a7-abb9-40e2-9744-7068195f5e00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487f04a7-abb9-40e2-9744-7068195f5e00-istiod-ca-cert\") pod \"487f04a7-abb9-40e2-9744-7068195f5e00\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " Nov 03 22:38:47.593678 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.593615 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-workload-socket\") pod \"487f04a7-abb9-40e2-9744-7068195f5e00\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " Nov 03 22:38:47.593678 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.593654 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/487f04a7-abb9-40e2-9744-7068195f5e00-istio-token\" (UniqueName: \"kubernetes.io/projected/487f04a7-abb9-40e2-9744-7068195f5e00-istio-token\") pod \"487f04a7-abb9-40e2-9744-7068195f5e00\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " Nov 03 22:38:47.594214 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.593784 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-istio-data\") pod \"487f04a7-abb9-40e2-9744-7068195f5e00\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " Nov 03 22:38:47.594214 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.593832 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-istio-envoy\") pod \"487f04a7-abb9-40e2-9744-7068195f5e00\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " Nov 03 22:38:47.594214 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.593873 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-credential-socket\") pod \"487f04a7-abb9-40e2-9744-7068195f5e00\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " Nov 03 22:38:47.594214 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.594110 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-istio-data" pod "487f04a7-abb9-40e2-9744-7068195f5e00" (UID: "487f04a7-abb9-40e2-9744-7068195f5e00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.594214 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.594153 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/487f04a7-abb9-40e2-9744-7068195f5e00-istiod-ca-cert" pod "487f04a7-abb9-40e2-9744-7068195f5e00" (UID: "487f04a7-abb9-40e2-9744-7068195f5e00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:47.594214 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.594182 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/487f04a7-abb9-40e2-9744-7068195f5e00-kube-api-access-md55q\" (UniqueName: \"kubernetes.io/projected/487f04a7-abb9-40e2-9744-7068195f5e00-kube-api-access-md55q\") pod \"487f04a7-abb9-40e2-9744-7068195f5e00\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " Nov 03 22:38:47.594537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.594383 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-workload-socket" pod "487f04a7-abb9-40e2-9744-7068195f5e00" (UID: "487f04a7-abb9-40e2-9744-7068195f5e00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.594537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.594395 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-credential-socket" pod "487f04a7-abb9-40e2-9744-7068195f5e00" (UID: "487f04a7-abb9-40e2-9744-7068195f5e00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.594656 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.594545 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-workload-certs\") pod \"487f04a7-abb9-40e2-9744-7068195f5e00\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " Nov 03 22:38:47.594716 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.594698 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/487f04a7-abb9-40e2-9744-7068195f5e00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487f04a7-abb9-40e2-9744-7068195f5e00-istio-podinfo\") pod \"487f04a7-abb9-40e2-9744-7068195f5e00\" (UID: \"487f04a7-abb9-40e2-9744-7068195f5e00\") " Nov 03 22:38:47.594844 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.594819 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-workload-certs" pod "487f04a7-abb9-40e2-9744-7068195f5e00" (UID: "487f04a7-abb9-40e2-9744-7068195f5e00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.596748 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.596010 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.596933 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.596917 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/487f04a7-abb9-40e2-9744-7068195f5e00-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.597029 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.597006 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-istio-envoy" pod "487f04a7-abb9-40e2-9744-7068195f5e00" (UID: "487f04a7-abb9-40e2-9744-7068195f5e00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:47.597126 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.597101 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487f04a7-abb9-40e2-9744-7068195f5e00-istio-token" pod "487f04a7-abb9-40e2-9744-7068195f5e00" (UID: "487f04a7-abb9-40e2-9744-7068195f5e00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:47.597185 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.597157 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/487f04a7-abb9-40e2-9744-7068195f5e00-istio-podinfo" pod "487f04a7-abb9-40e2-9744-7068195f5e00" (UID: "487f04a7-abb9-40e2-9744-7068195f5e00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:38:47.597238 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.597185 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487f04a7-abb9-40e2-9744-7068195f5e00-kube-api-access-md55q" pod "487f04a7-abb9-40e2-9744-7068195f5e00" (UID: "487f04a7-abb9-40e2-9744-7068195f5e00"). InnerVolumeSpecName "kube-api-access-md55q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:47.597331 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.597304 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.597449 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.597335 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.597449 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.597350 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.631460 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.631411 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk"] Nov 03 22:38:47.635029 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.635002 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-zpjrk"] Nov 03 22:38:47.699052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.699013 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/487f04a7-abb9-40e2-9744-7068195f5e00-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.699052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.699051 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/487f04a7-abb9-40e2-9744-7068195f5e00-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.699052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.699074 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/487f04a7-abb9-40e2-9744-7068195f5e00-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.699326 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.699083 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md55q\" (UniqueName: \"kubernetes.io/projected/487f04a7-abb9-40e2-9744-7068195f5e00-kube-api-access-md55q\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:38:47.798500 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.798457 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982"] Nov 03 22:38:47.804125 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:47.804054 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-sq982"] Nov 03 22:38:48.320952 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:48.320924 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" Nov 03 22:38:48.321416 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:48.320918 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp" event={"ID":"487f04a7-abb9-40e2-9744-7068195f5e00","Type":"ContainerDied","Data":"b081394fb0ad758e5468a88e80d8b8bbdf8452ba598fcd17cd5cb74d4dce4e0e"} Nov 03 22:38:48.321416 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:48.321101 5084 scope.go:122] "RemoveContainer" containerID="19c33cfca9ed5d86c52f78b598c2ac06301559c652e23397ae53c5ef8e327ba1" Nov 03 22:38:48.490263 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:48.490220 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp"] Nov 03 22:38:48.495637 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:48.495602 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-d7hrp"] Nov 03 22:38:48.630481 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:48.630393 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="487f04a7-abb9-40e2-9744-7068195f5e00" path="/var/lib/kubelet/pods/487f04a7-abb9-40e2-9744-7068195f5e00/volumes" Nov 03 22:38:48.631079 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:48.631040 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ca4f5ec-9244-447b-ba40-91409196e307" path="/var/lib/kubelet/pods/6ca4f5ec-9244-447b-ba40-91409196e307/volumes" Nov 03 22:38:48.631641 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:48.631619 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" path="/var/lib/kubelet/pods/93bd9f01-32b4-42ae-9ef8-f0bef05b18b2/volumes" Nov 03 22:38:48.632542 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:48.632518 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" path="/var/lib/kubelet/pods/cfc48515-bea2-4a71-8d48-c6000b2a46d9/volumes" Nov 03 22:38:55.612180 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:55.612142 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d"] Nov 03 22:38:55.612614 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:55.612372 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" podUID="462d5bbd-4b7b-4c70-857a-99964a7bdef3" containerName="istio-proxy" containerID="cri-o://f1fd7970e135d1b999ae70429acb4defb6bf308e838d1a88b34240e110786151" gracePeriod=30 Nov 03 22:38:55.987817 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:55.987687 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc"] Nov 03 22:38:55.987989 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:55.987922 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" podUID="231a9925-fa0d-4a64-b4e6-cdeaf59d44be" containerName="istio-proxy" containerID="cri-o://58dae898ea1d44e88837d3579719a44179245956841753c3b907bedb73cfc496" gracePeriod=30 Nov 03 22:38:56.241326 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:38:56.241230 5084 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb4c5a2_c6c1_4be9_9656_5eecee18ce49.slice/crio-conmon-e87355f8b45c32ef5cffc001590054967f8e05bfdadba0cca1f87ca6cb15d7b1.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:38:56.354909 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:56.354860 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:38:56.355372 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:56.354947 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" podUID="231a9925-fa0d-4a64-b4e6-cdeaf59d44be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:38:56.388967 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:56.388933 5084 generic.go:356] "Generic (PLEG): container finished" podID="5bb4c5a2-c6c1-4be9-9656-5eecee18ce49" containerID="e87355f8b45c32ef5cffc001590054967f8e05bfdadba0cca1f87ca6cb15d7b1" exitCode=1 Nov 03 22:38:56.389198 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:56.389011 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" event={"ID":"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49","Type":"ContainerDied","Data":"e87355f8b45c32ef5cffc001590054967f8e05bfdadba0cca1f87ca6cb15d7b1"} Nov 03 22:38:56.389198 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:56.389165 5084 scope.go:122] "RemoveContainer" containerID="cc06bd7900e1f2ecc00f506ee29760e024ffdbbb90c42425711b77a816a4819e" Nov 03 22:38:56.389699 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:56.389673 5084 scope.go:122] "RemoveContainer" containerID="e87355f8b45c32ef5cffc001590054967f8e05bfdadba0cca1f87ca6cb15d7b1" Nov 03 22:38:57.291157 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:57.291125 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 22:38:57.400368 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:57.400322 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" event={"ID":"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49","Type":"ContainerStarted","Data":"80574e6f20cc41d8e49c6261bc33f056fc1520d8431785ba6e6188c690e790a6"} Nov 03 22:38:57.400610 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:38:57.400571 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:39:00.934634 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.934591 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:39:00.995445 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.995406 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462d5bbd-4b7b-4c70-857a-99964a7bdef3-kube-api-access-xgclh\" (UniqueName: \"kubernetes.io/projected/462d5bbd-4b7b-4c70-857a-99964a7bdef3-kube-api-access-xgclh\") pod \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " Nov 03 22:39:00.995650 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.995455 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-token\" (UniqueName: \"kubernetes.io/projected/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-token\") pod \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " Nov 03 22:39:00.995650 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.995483 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-podinfo\") pod \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " Nov 03 22:39:00.995650 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.995594 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-workload-socket\") pod \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " Nov 03 22:39:00.995835 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.995680 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-workload-certs\") pod \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " Nov 03 22:39:00.995835 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.995719 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istiod-ca-cert\") pod \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " Nov 03 22:39:00.995835 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.995796 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-envoy\") pod \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " Nov 03 22:39:00.995835 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.995824 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-credential-socket\") pod \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " Nov 03 22:39:00.996104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.995932 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-workload-socket" pod "462d5bbd-4b7b-4c70-857a-99964a7bdef3" (UID: "462d5bbd-4b7b-4c70-857a-99964a7bdef3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:00.996104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.995970 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-data\") pod \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\" (UID: \"462d5bbd-4b7b-4c70-857a-99964a7bdef3\") " Nov 03 22:39:00.996104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.996032 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-workload-certs" pod "462d5bbd-4b7b-4c70-857a-99964a7bdef3" (UID: "462d5bbd-4b7b-4c70-857a-99964a7bdef3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:00.996375 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.996301 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istiod-ca-cert" pod "462d5bbd-4b7b-4c70-857a-99964a7bdef3" (UID: "462d5bbd-4b7b-4c70-857a-99964a7bdef3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:00.996572 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.996545 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-credential-socket" pod "462d5bbd-4b7b-4c70-857a-99964a7bdef3" (UID: "462d5bbd-4b7b-4c70-857a-99964a7bdef3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:00.996682 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.996656 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-data" pod "462d5bbd-4b7b-4c70-857a-99964a7bdef3" (UID: "462d5bbd-4b7b-4c70-857a-99964a7bdef3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:00.997050 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.997024 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:00.997178 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.997056 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:00.997178 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.997089 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:00.997178 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.997105 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:00.997178 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.997120 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:00.998777 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:00.998742 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462d5bbd-4b7b-4c70-857a-99964a7bdef3-kube-api-access-xgclh" pod "462d5bbd-4b7b-4c70-857a-99964a7bdef3" (UID: "462d5bbd-4b7b-4c70-857a-99964a7bdef3"). InnerVolumeSpecName "kube-api-access-xgclh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:01.000263 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.000222 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-podinfo" pod "462d5bbd-4b7b-4c70-857a-99964a7bdef3" (UID: "462d5bbd-4b7b-4c70-857a-99964a7bdef3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:39:01.001691 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.001663 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-envoy" pod "462d5bbd-4b7b-4c70-857a-99964a7bdef3" (UID: "462d5bbd-4b7b-4c70-857a-99964a7bdef3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:01.001842 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.001823 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-token" pod "462d5bbd-4b7b-4c70-857a-99964a7bdef3" (UID: "462d5bbd-4b7b-4c70-857a-99964a7bdef3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:01.098337 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.098299 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgclh\" (UniqueName: \"kubernetes.io/projected/462d5bbd-4b7b-4c70-857a-99964a7bdef3-kube-api-access-xgclh\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.098337 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.098335 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.098337 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.098345 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.098579 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.098353 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/462d5bbd-4b7b-4c70-857a-99964a7bdef3-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.282405 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.282367 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:39:01.406426 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.406387 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istiod-ca-cert\") pod \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " Nov 03 22:39:01.406675 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.406455 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-envoy\") pod \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " Nov 03 22:39:01.406675 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.406495 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-kube-api-access-4n5sb\" (UniqueName: \"kubernetes.io/projected/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-kube-api-access-4n5sb\") pod \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " Nov 03 22:39:01.406675 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.406559 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-token\" (UniqueName: \"kubernetes.io/projected/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-token\") pod \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " Nov 03 22:39:01.406675 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.406665 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-credential-socket\") pod \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " Nov 03 22:39:01.406941 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.406708 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-data\") pod \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " Nov 03 22:39:01.406941 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.406821 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-podinfo\") pod \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " Nov 03 22:39:01.406941 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.406872 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-workload-certs\") pod \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " Nov 03 22:39:01.406941 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.406872 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istiod-ca-cert" pod "231a9925-fa0d-4a64-b4e6-cdeaf59d44be" (UID: "231a9925-fa0d-4a64-b4e6-cdeaf59d44be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:01.406941 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.406898 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-workload-socket\") pod \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\" (UID: \"231a9925-fa0d-4a64-b4e6-cdeaf59d44be\") " Nov 03 22:39:01.407242 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.407058 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-credential-socket" pod "231a9925-fa0d-4a64-b4e6-cdeaf59d44be" (UID: "231a9925-fa0d-4a64-b4e6-cdeaf59d44be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:01.407412 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.407389 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-workload-socket" pod "231a9925-fa0d-4a64-b4e6-cdeaf59d44be" (UID: "231a9925-fa0d-4a64-b4e6-cdeaf59d44be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:01.408018 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.407696 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.408018 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.407725 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.408018 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.407740 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.408018 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.407852 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-data" pod "231a9925-fa0d-4a64-b4e6-cdeaf59d44be" (UID: "231a9925-fa0d-4a64-b4e6-cdeaf59d44be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:01.408018 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.408002 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-workload-certs" pod "231a9925-fa0d-4a64-b4e6-cdeaf59d44be" (UID: "231a9925-fa0d-4a64-b4e6-cdeaf59d44be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:01.409228 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.409199 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-envoy" pod "231a9925-fa0d-4a64-b4e6-cdeaf59d44be" (UID: "231a9925-fa0d-4a64-b4e6-cdeaf59d44be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:01.409627 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.409601 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-token" pod "231a9925-fa0d-4a64-b4e6-cdeaf59d44be" (UID: "231a9925-fa0d-4a64-b4e6-cdeaf59d44be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:01.409786 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.409760 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-podinfo" pod "231a9925-fa0d-4a64-b4e6-cdeaf59d44be" (UID: "231a9925-fa0d-4a64-b4e6-cdeaf59d44be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:39:01.410154 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.410135 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-kube-api-access-4n5sb" pod "231a9925-fa0d-4a64-b4e6-cdeaf59d44be" (UID: "231a9925-fa0d-4a64-b4e6-cdeaf59d44be"). InnerVolumeSpecName "kube-api-access-4n5sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:01.437671 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.437562 5084 generic.go:356] "Generic (PLEG): container finished" podID="231a9925-fa0d-4a64-b4e6-cdeaf59d44be" containerID="58dae898ea1d44e88837d3579719a44179245956841753c3b907bedb73cfc496" exitCode=0 Nov 03 22:39:01.437671 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.437629 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" Nov 03 22:39:01.438012 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.437665 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" event={"ID":"231a9925-fa0d-4a64-b4e6-cdeaf59d44be","Type":"ContainerDied","Data":"58dae898ea1d44e88837d3579719a44179245956841753c3b907bedb73cfc496"} Nov 03 22:39:01.438012 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.437723 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc" event={"ID":"231a9925-fa0d-4a64-b4e6-cdeaf59d44be","Type":"ContainerDied","Data":"aad23be0317ec6b7436f6f301edaeab505da363583c4f85b0620de458baa5fa3"} Nov 03 22:39:01.438012 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.437755 5084 scope.go:122] "RemoveContainer" containerID="58dae898ea1d44e88837d3579719a44179245956841753c3b907bedb73cfc496" Nov 03 22:39:01.446504 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.446102 5084 generic.go:356] "Generic (PLEG): container finished" podID="462d5bbd-4b7b-4c70-857a-99964a7bdef3" containerID="f1fd7970e135d1b999ae70429acb4defb6bf308e838d1a88b34240e110786151" exitCode=0 Nov 03 22:39:01.446504 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.446244 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" event={"ID":"462d5bbd-4b7b-4c70-857a-99964a7bdef3","Type":"ContainerDied","Data":"f1fd7970e135d1b999ae70429acb4defb6bf308e838d1a88b34240e110786151"} Nov 03 22:39:01.446504 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.446326 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" event={"ID":"462d5bbd-4b7b-4c70-857a-99964a7bdef3","Type":"ContainerDied","Data":"228c44b501e112ea85fa9d70508226f23e88d9eb9962fa85b7f127d9e51be0ad"} Nov 03 22:39:01.446504 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.446457 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d" Nov 03 22:39:01.461444 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.461416 5084 scope.go:122] "RemoveContainer" containerID="58dae898ea1d44e88837d3579719a44179245956841753c3b907bedb73cfc496" Nov 03 22:39:01.461868 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:39:01.461832 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58dae898ea1d44e88837d3579719a44179245956841753c3b907bedb73cfc496\": container with ID starting with 58dae898ea1d44e88837d3579719a44179245956841753c3b907bedb73cfc496 not found: ID does not exist" containerID="58dae898ea1d44e88837d3579719a44179245956841753c3b907bedb73cfc496" Nov 03 22:39:01.461980 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.461872 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58dae898ea1d44e88837d3579719a44179245956841753c3b907bedb73cfc496"} err="failed to get container status \"58dae898ea1d44e88837d3579719a44179245956841753c3b907bedb73cfc496\": rpc error: code = NotFound desc = could not find container \"58dae898ea1d44e88837d3579719a44179245956841753c3b907bedb73cfc496\": container with ID starting with 58dae898ea1d44e88837d3579719a44179245956841753c3b907bedb73cfc496 not found: ID does not exist" Nov 03 22:39:01.461980 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.461898 5084 scope.go:122] "RemoveContainer" containerID="f1fd7970e135d1b999ae70429acb4defb6bf308e838d1a88b34240e110786151" Nov 03 22:39:01.474028 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.473997 5084 scope.go:122] "RemoveContainer" containerID="f1fd7970e135d1b999ae70429acb4defb6bf308e838d1a88b34240e110786151" Nov 03 22:39:01.474410 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:39:01.474382 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1fd7970e135d1b999ae70429acb4defb6bf308e838d1a88b34240e110786151\": container with ID starting with f1fd7970e135d1b999ae70429acb4defb6bf308e838d1a88b34240e110786151 not found: ID does not exist" containerID="f1fd7970e135d1b999ae70429acb4defb6bf308e838d1a88b34240e110786151" Nov 03 22:39:01.474526 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.474418 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fd7970e135d1b999ae70429acb4defb6bf308e838d1a88b34240e110786151"} err="failed to get container status \"f1fd7970e135d1b999ae70429acb4defb6bf308e838d1a88b34240e110786151\": rpc error: code = NotFound desc = could not find container \"f1fd7970e135d1b999ae70429acb4defb6bf308e838d1a88b34240e110786151\": container with ID starting with f1fd7970e135d1b999ae70429acb4defb6bf308e838d1a88b34240e110786151 not found: ID does not exist" Nov 03 22:39:01.509179 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.509136 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.509179 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.509174 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n5sb\" (UniqueName: \"kubernetes.io/projected/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-kube-api-access-4n5sb\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.509179 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.509188 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.509457 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.509197 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.509457 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.509206 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.509457 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.509216 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231a9925-fa0d-4a64-b4e6-cdeaf59d44be-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:01.607553 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.607496 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc"] Nov 03 22:39:01.611836 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.611797 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-flqgc"] Nov 03 22:39:01.721240 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.721155 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d"] Nov 03 22:39:01.725245 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:01.725212 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-b7h2d"] Nov 03 22:39:02.630232 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:02.630200 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231a9925-fa0d-4a64-b4e6-cdeaf59d44be" path="/var/lib/kubelet/pods/231a9925-fa0d-4a64-b4e6-cdeaf59d44be/volumes" Nov 03 22:39:02.630746 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:02.630730 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="462d5bbd-4b7b-4c70-857a-99964a7bdef3" path="/var/lib/kubelet/pods/462d5bbd-4b7b-4c70-857a-99964a7bdef3/volumes" Nov 03 22:39:08.411032 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:08.410993 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:39:12.026333 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:39:12.026295 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:39:36.018839 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.018744 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr"] Nov 03 22:39:36.019460 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019442 5084 cpu_manager.go:414] "containerMap: removing container" podUID="50e61dd6-336d-48cb-bda1-7ee80c1e27b9" containerName="istio-proxy" Nov 03 22:39:36.019509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019461 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50e61dd6-336d-48cb-bda1-7ee80c1e27b9" containerName="istio-proxy" Nov 03 22:39:36.019509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019485 5084 cpu_manager.go:414] "containerMap: removing container" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerName="istio-proxy" Nov 03 22:39:36.019509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019490 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerName="istio-proxy" Nov 03 22:39:36.019509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019498 5084 cpu_manager.go:414] "containerMap: removing container" podUID="6ca4f5ec-9244-447b-ba40-91409196e307" containerName="istio-proxy" Nov 03 22:39:36.019509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019506 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ca4f5ec-9244-447b-ba40-91409196e307" containerName="istio-proxy" Nov 03 22:39:36.019667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019520 5084 cpu_manager.go:414] "containerMap: removing container" podUID="462d5bbd-4b7b-4c70-857a-99964a7bdef3" containerName="istio-proxy" Nov 03 22:39:36.019667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019526 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462d5bbd-4b7b-4c70-857a-99964a7bdef3" containerName="istio-proxy" Nov 03 22:39:36.019667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019537 5084 cpu_manager.go:414] "containerMap: removing container" podUID="487f04a7-abb9-40e2-9744-7068195f5e00" containerName="istio-proxy" Nov 03 22:39:36.019667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019542 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="487f04a7-abb9-40e2-9744-7068195f5e00" containerName="istio-proxy" Nov 03 22:39:36.019667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019567 5084 cpu_manager.go:414] "containerMap: removing container" podUID="93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" containerName="istio-proxy" Nov 03 22:39:36.019667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019572 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" containerName="istio-proxy" Nov 03 22:39:36.019667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019587 5084 cpu_manager.go:414] "containerMap: removing container" podUID="231a9925-fa0d-4a64-b4e6-cdeaf59d44be" containerName="istio-proxy" Nov 03 22:39:36.019667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019591 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231a9925-fa0d-4a64-b4e6-cdeaf59d44be" containerName="istio-proxy" Nov 03 22:39:36.019900 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019691 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="50e61dd6-336d-48cb-bda1-7ee80c1e27b9" containerName="istio-proxy" Nov 03 22:39:36.019900 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019700 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc48515-bea2-4a71-8d48-c6000b2a46d9" containerName="istio-proxy" Nov 03 22:39:36.019900 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019705 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ca4f5ec-9244-447b-ba40-91409196e307" containerName="istio-proxy" Nov 03 22:39:36.019900 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019712 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="462d5bbd-4b7b-4c70-857a-99964a7bdef3" containerName="istio-proxy" Nov 03 22:39:36.019900 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019718 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="487f04a7-abb9-40e2-9744-7068195f5e00" containerName="istio-proxy" Nov 03 22:39:36.019900 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019727 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="93bd9f01-32b4-42ae-9ef8-f0bef05b18b2" containerName="istio-proxy" Nov 03 22:39:36.019900 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.019733 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="231a9925-fa0d-4a64-b4e6-cdeaf59d44be" containerName="istio-proxy" Nov 03 22:39:36.023552 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.023524 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" Nov 03 22:39:36.076138 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.076094 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr"] Nov 03 22:39:36.122402 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.122357 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-e7d36a-serving-cert\"" Nov 03 22:39:36.188553 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.188516 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03858435-0753-4dd7-9a58-ad06f4dbc39c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr\" (UID: \"03858435-0753-4dd7-9a58-ad06f4dbc39c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" Nov 03 22:39:36.188969 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.188943 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb9jd\" (UniqueName: \"kubernetes.io/projected/03858435-0753-4dd7-9a58-ad06f4dbc39c-kube-api-access-hb9jd\") pod \"maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr\" (UID: \"03858435-0753-4dd7-9a58-ad06f4dbc39c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" Nov 03 22:39:36.189354 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.189284 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03858435-0753-4dd7-9a58-ad06f4dbc39c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr\" (UID: \"03858435-0753-4dd7-9a58-ad06f4dbc39c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" Nov 03 22:39:36.189557 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.189539 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03858435-0753-4dd7-9a58-ad06f4dbc39c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr\" (UID: \"03858435-0753-4dd7-9a58-ad06f4dbc39c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" Nov 03 22:39:36.318144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.305148 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03858435-0753-4dd7-9a58-ad06f4dbc39c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr\" (UID: \"03858435-0753-4dd7-9a58-ad06f4dbc39c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" Nov 03 22:39:36.318144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.305272 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9jd\" (UniqueName: \"kubernetes.io/projected/03858435-0753-4dd7-9a58-ad06f4dbc39c-kube-api-access-hb9jd\") pod \"maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr\" (UID: \"03858435-0753-4dd7-9a58-ad06f4dbc39c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" Nov 03 22:39:36.318144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.305363 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03858435-0753-4dd7-9a58-ad06f4dbc39c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr\" (UID: \"03858435-0753-4dd7-9a58-ad06f4dbc39c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" Nov 03 22:39:36.318144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.305401 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03858435-0753-4dd7-9a58-ad06f4dbc39c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr\" (UID: \"03858435-0753-4dd7-9a58-ad06f4dbc39c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" Nov 03 22:39:36.329890 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.323605 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03858435-0753-4dd7-9a58-ad06f4dbc39c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr\" (UID: \"03858435-0753-4dd7-9a58-ad06f4dbc39c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" Nov 03 22:39:36.351843 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.351790 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03858435-0753-4dd7-9a58-ad06f4dbc39c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr\" (UID: \"03858435-0753-4dd7-9a58-ad06f4dbc39c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" Nov 03 22:39:36.380857 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.380803 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03858435-0753-4dd7-9a58-ad06f4dbc39c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr\" (UID: \"03858435-0753-4dd7-9a58-ad06f4dbc39c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" Nov 03 22:39:36.385190 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.385156 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb9jd\" (UniqueName: \"kubernetes.io/projected/03858435-0753-4dd7-9a58-ad06f4dbc39c-kube-api-access-hb9jd\") pod \"maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr\" (UID: \"03858435-0753-4dd7-9a58-ad06f4dbc39c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" Nov 03 22:39:36.640644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:36.640547 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" Nov 03 22:39:37.031801 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:37.031756 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr"] Nov 03 22:39:37.032747 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:39:37.032714 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03858435_0753_4dd7_9a58_ad06f4dbc39c.slice/crio-31997110660c0fda519e98dfcad5041a20ae9222f8fb63185d66c6174255e551 WatchSource:0}: Error finding container 31997110660c0fda519e98dfcad5041a20ae9222f8fb63185d66c6174255e551: Status 404 returned error can't find the container with id 31997110660c0fda519e98dfcad5041a20ae9222f8fb63185d66c6174255e551 Nov 03 22:39:37.774105 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:37.774055 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" event={"ID":"03858435-0753-4dd7-9a58-ad06f4dbc39c","Type":"ContainerStarted","Data":"31997110660c0fda519e98dfcad5041a20ae9222f8fb63185d66c6174255e551"} Nov 03 22:39:38.783714 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:38.783665 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" event={"ID":"03858435-0753-4dd7-9a58-ad06f4dbc39c","Type":"ContainerStarted","Data":"310eb825514cd7e6fcdedc7310daf737eaa6247e913db0fb7bb11095bd10852b"} Nov 03 22:39:38.784142 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:38.783743 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" Nov 03 22:39:38.905654 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:38.905194 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" podStartSLOduration=2.998257259 podStartE2EDuration="3.905173374s" totalImagesPullingTime="906.916115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:38.903820963 +0000 UTC m=+1946.497852937" watchObservedRunningTime="2026-11-03 22:39:38.905173374 +0000 UTC m=+1946.499205351" Nov 03 22:39:40.701904 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:40.701858 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq"] Nov 03 22:39:40.702485 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:40.702113 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" podUID="4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61" containerName="maas-api" containerID="cri-o://8c440dda66ef41bdfbded79f3fc4e35791bc1a32590d846063519dc39ba50100" gracePeriod=30 Nov 03 22:39:41.152503 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.152466 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" Nov 03 22:39:41.256089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.253643 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-metrics-certs\") pod \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\" (UID: \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\") " Nov 03 22:39:41.256089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.253722 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-openshift-service-ca\") pod \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\" (UID: \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\") " Nov 03 22:39:41.256089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.253870 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-kube-api-access-dj468\" (UniqueName: \"kubernetes.io/projected/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-kube-api-access-dj468\") pod \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\" (UID: \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\") " Nov 03 22:39:41.256089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.254043 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-maas-api-tls\") pod \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\" (UID: \"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61\") " Nov 03 22:39:41.261092 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.258200 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-maas-api-tls" pod "4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61" (UID: "4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:39:41.261762 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.261707 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-openshift-service-ca" pod "4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61" (UID: "4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:41.272667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.272605 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-kube-api-access-dj468" pod "4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61" (UID: "4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61"). InnerVolumeSpecName "kube-api-access-dj468". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:41.286142 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.284748 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-metrics-certs" pod "4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61" (UID: "4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:39:41.356160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.356029 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj468\" (UniqueName: \"kubernetes.io/projected/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-kube-api-access-dj468\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:41.356160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.356084 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:41.356160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.356101 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:41.356160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.356113 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:39:41.935994 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.933793 5084 generic.go:356] "Generic (PLEG): container finished" podID="4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61" containerID="8c440dda66ef41bdfbded79f3fc4e35791bc1a32590d846063519dc39ba50100" exitCode=0 Nov 03 22:39:41.935994 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.933879 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" event={"ID":"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61","Type":"ContainerDied","Data":"8c440dda66ef41bdfbded79f3fc4e35791bc1a32590d846063519dc39ba50100"} Nov 03 22:39:41.935994 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.934232 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" event={"ID":"4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61","Type":"ContainerDied","Data":"daa6e344030d472dac56c90ff71e0ad6f41da40fc7274cdef56ba2fe8e4f4420"} Nov 03 22:39:41.935994 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.934263 5084 scope.go:122] "RemoveContainer" containerID="8c440dda66ef41bdfbded79f3fc4e35791bc1a32590d846063519dc39ba50100" Nov 03 22:39:41.935994 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.934563 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq" Nov 03 22:39:41.947184 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.947149 5084 scope.go:122] "RemoveContainer" containerID="8c440dda66ef41bdfbded79f3fc4e35791bc1a32590d846063519dc39ba50100" Nov 03 22:39:41.947573 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:39:41.947540 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c440dda66ef41bdfbded79f3fc4e35791bc1a32590d846063519dc39ba50100\": container with ID starting with 8c440dda66ef41bdfbded79f3fc4e35791bc1a32590d846063519dc39ba50100 not found: ID does not exist" containerID="8c440dda66ef41bdfbded79f3fc4e35791bc1a32590d846063519dc39ba50100" Nov 03 22:39:41.947697 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:41.947578 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c440dda66ef41bdfbded79f3fc4e35791bc1a32590d846063519dc39ba50100"} err="failed to get container status \"8c440dda66ef41bdfbded79f3fc4e35791bc1a32590d846063519dc39ba50100\": rpc error: code = NotFound desc = could not find container \"8c440dda66ef41bdfbded79f3fc4e35791bc1a32590d846063519dc39ba50100\": container with ID starting with 8c440dda66ef41bdfbded79f3fc4e35791bc1a32590d846063519dc39ba50100 not found: ID does not exist" Nov 03 22:39:42.025321 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:39:42.025277 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:39:42.104425 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:42.104389 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq"] Nov 03 22:39:42.108746 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:42.108714 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-2fvdq"] Nov 03 22:39:42.630445 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:42.630410 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61" path="/var/lib/kubelet/pods/4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61/volumes" Nov 03 22:39:44.797912 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:44.797882 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" Nov 03 22:39:47.334765 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.334727 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2"] Nov 03 22:39:47.335429 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.335413 5084 cpu_manager.go:414] "containerMap: removing container" podUID="4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61" containerName="maas-api" Nov 03 22:39:47.335429 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.335430 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61" containerName="maas-api" Nov 03 22:39:47.335591 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.335580 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="4872ed1e-82e9-412c-a5c5-1ae3a4cf4a61" containerName="maas-api" Nov 03 22:39:47.340895 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.340871 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" Nov 03 22:39:47.396249 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.396210 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2"] Nov 03 22:39:47.440226 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.439629 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-345da9-dockercfg-nph2g\"" Nov 03 22:39:47.440226 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.439754 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-345da9\"" Nov 03 22:39:47.566102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.546846 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a565b6a-0d39-4a6b-bfeb-af467508b071-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2\" (UID: \"9a565b6a-0d39-4a6b-bfeb-af467508b071\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" Nov 03 22:39:47.566825 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.566785 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkt27\" (UniqueName: \"kubernetes.io/projected/9a565b6a-0d39-4a6b-bfeb-af467508b071-kube-api-access-gkt27\") pod \"payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2\" (UID: \"9a565b6a-0d39-4a6b-bfeb-af467508b071\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" Nov 03 22:39:47.736184 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.734311 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a565b6a-0d39-4a6b-bfeb-af467508b071-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2\" (UID: \"9a565b6a-0d39-4a6b-bfeb-af467508b071\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" Nov 03 22:39:47.736184 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.734840 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkt27\" (UniqueName: \"kubernetes.io/projected/9a565b6a-0d39-4a6b-bfeb-af467508b071-kube-api-access-gkt27\") pod \"payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2\" (UID: \"9a565b6a-0d39-4a6b-bfeb-af467508b071\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" Nov 03 22:39:47.737001 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.736957 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a565b6a-0d39-4a6b-bfeb-af467508b071-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2\" (UID: \"9a565b6a-0d39-4a6b-bfeb-af467508b071\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" Nov 03 22:39:47.741168 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.741131 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6"] Nov 03 22:39:47.751819 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.751786 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6"] Nov 03 22:39:47.752185 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.752164 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6" Nov 03 22:39:47.792780 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.792749 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkt27\" (UniqueName: \"kubernetes.io/projected/9a565b6a-0d39-4a6b-bfeb-af467508b071-kube-api-access-gkt27\") pod \"payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2\" (UID: \"9a565b6a-0d39-4a6b-bfeb-af467508b071\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" Nov 03 22:39:47.836928 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.836881 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dab1e4-6fcb-49a2-b04d-a803633c232d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6\" (UID: \"e9dab1e4-6fcb-49a2-b04d-a803633c232d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6" Nov 03 22:39:47.836928 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.836929 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxwhq\" (UniqueName: \"kubernetes.io/projected/e9dab1e4-6fcb-49a2-b04d-a803633c232d-kube-api-access-hxwhq\") pod \"payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6\" (UID: \"e9dab1e4-6fcb-49a2-b04d-a803633c232d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6" Nov 03 22:39:47.938436 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.938394 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dab1e4-6fcb-49a2-b04d-a803633c232d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6\" (UID: \"e9dab1e4-6fcb-49a2-b04d-a803633c232d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6" Nov 03 22:39:47.938436 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.938441 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxwhq\" (UniqueName: \"kubernetes.io/projected/e9dab1e4-6fcb-49a2-b04d-a803633c232d-kube-api-access-hxwhq\") pod \"payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6\" (UID: \"e9dab1e4-6fcb-49a2-b04d-a803633c232d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6" Nov 03 22:39:47.939275 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.939252 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dab1e4-6fcb-49a2-b04d-a803633c232d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6\" (UID: \"e9dab1e4-6fcb-49a2-b04d-a803633c232d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6" Nov 03 22:39:47.958026 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.957978 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" Nov 03 22:39:47.996275 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:47.996185 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxwhq\" (UniqueName: \"kubernetes.io/projected/e9dab1e4-6fcb-49a2-b04d-a803633c232d-kube-api-access-hxwhq\") pod \"payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6\" (UID: \"e9dab1e4-6fcb-49a2-b04d-a803633c232d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6" Nov 03 22:39:48.076114 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:48.076079 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6" Nov 03 22:39:48.346828 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:48.346786 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2"] Nov 03 22:39:48.347351 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:39:48.347327 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a565b6a_0d39_4a6b_bfeb_af467508b071.slice/crio-e0ec274743f7bdbd8144818372ee08869951401d8dd7fa342503a0ed9040d919 WatchSource:0}: Error finding container e0ec274743f7bdbd8144818372ee08869951401d8dd7fa342503a0ed9040d919: Status 404 returned error can't find the container with id e0ec274743f7bdbd8144818372ee08869951401d8dd7fa342503a0ed9040d919 Nov 03 22:39:48.464864 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:39:48.464833 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9dab1e4_6fcb_49a2_b04d_a803633c232d.slice/crio-aa2bf90154563b62448cf947e96d91a9ca7fdfa40fcb4f25ead0dec9139671ff WatchSource:0}: Error finding container aa2bf90154563b62448cf947e96d91a9ca7fdfa40fcb4f25ead0dec9139671ff: Status 404 returned error can't find the container with id aa2bf90154563b62448cf947e96d91a9ca7fdfa40fcb4f25ead0dec9139671ff Nov 03 22:39:48.466852 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:48.466818 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6"] Nov 03 22:39:49.002604 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:49.002566 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6" event={"ID":"e9dab1e4-6fcb-49a2-b04d-a803633c232d","Type":"ContainerStarted","Data":"f4e4009578c88f3c18c4bc9dcb2ba798362624cab9e2dc6727e1d81776480a53"} Nov 03 22:39:49.002838 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:49.002626 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6" event={"ID":"e9dab1e4-6fcb-49a2-b04d-a803633c232d","Type":"ContainerStarted","Data":"aa2bf90154563b62448cf947e96d91a9ca7fdfa40fcb4f25ead0dec9139671ff"} Nov 03 22:39:49.002838 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:49.002667 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6" Nov 03 22:39:49.004564 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:49.004537 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" event={"ID":"9a565b6a-0d39-4a6b-bfeb-af467508b071","Type":"ContainerStarted","Data":"da25510a51ad94bacab8efaf6550928926e4eb81e4fe306050cbf6b31a633922"} Nov 03 22:39:49.004688 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:49.004593 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" event={"ID":"9a565b6a-0d39-4a6b-bfeb-af467508b071","Type":"ContainerStarted","Data":"e0ec274743f7bdbd8144818372ee08869951401d8dd7fa342503a0ed9040d919"} Nov 03 22:39:49.004688 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:49.004614 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" Nov 03 22:39:49.123280 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:49.123218 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6" podStartSLOduration=2.123200325 podStartE2EDuration="2.123200325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:49.118692615 +0000 UTC m=+1956.712724602" watchObservedRunningTime="2026-11-03 22:39:49.123200325 +0000 UTC m=+1956.717232298" Nov 03 22:39:49.252109 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:49.250602 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" podStartSLOduration=2.250576533 podStartE2EDuration="2.250576533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:49.234553678 +0000 UTC m=+1956.828585655" watchObservedRunningTime="2026-11-03 22:39:49.250576533 +0000 UTC m=+1956.844608508" Nov 03 22:39:54.727352 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:54.727281 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26"] Nov 03 22:39:54.734524 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:54.733540 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26"] Nov 03 22:39:54.734524 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:54.733673 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26" Nov 03 22:39:54.846389 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:54.845311 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-2a6801\"" Nov 03 22:39:54.846389 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:54.845353 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-2a6801-dockercfg-tmfpw\"" Nov 03 22:39:54.954564 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:54.904689 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr5st\" (UniqueName: \"kubernetes.io/projected/6a069ee1-c643-4a4a-8632-6e02db66d490-kube-api-access-xr5st\") pod \"payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26\" (UID: \"6a069ee1-c643-4a4a-8632-6e02db66d490\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26" Nov 03 22:39:54.954564 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:54.929208 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s"] Nov 03 22:39:54.954564 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:54.904782 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a069ee1-c643-4a4a-8632-6e02db66d490-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26\" (UID: \"6a069ee1-c643-4a4a-8632-6e02db66d490\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26" Nov 03 22:39:54.962770 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:54.962686 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s" Nov 03 22:39:54.968485 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:54.968126 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s"] Nov 03 22:39:55.055719 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:55.055681 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr5st\" (UniqueName: \"kubernetes.io/projected/6a069ee1-c643-4a4a-8632-6e02db66d490-kube-api-access-xr5st\") pod \"payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26\" (UID: \"6a069ee1-c643-4a4a-8632-6e02db66d490\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26" Nov 03 22:39:55.055944 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:55.055737 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a069ee1-c643-4a4a-8632-6e02db66d490-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26\" (UID: \"6a069ee1-c643-4a4a-8632-6e02db66d490\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26" Nov 03 22:39:55.056777 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:55.056750 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a069ee1-c643-4a4a-8632-6e02db66d490-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26\" (UID: \"6a069ee1-c643-4a4a-8632-6e02db66d490\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26" Nov 03 22:39:55.113986 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:55.113957 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr5st\" (UniqueName: \"kubernetes.io/projected/6a069ee1-c643-4a4a-8632-6e02db66d490-kube-api-access-xr5st\") pod \"payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26\" (UID: \"6a069ee1-c643-4a4a-8632-6e02db66d490\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26" Nov 03 22:39:55.157384 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:55.157338 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2778fdc5-5b89-4783-856c-4e26b7fe30f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s\" (UID: \"2778fdc5-5b89-4783-856c-4e26b7fe30f6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s" Nov 03 22:39:55.157598 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:55.157535 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjqmx\" (UniqueName: \"kubernetes.io/projected/2778fdc5-5b89-4783-856c-4e26b7fe30f6-kube-api-access-hjqmx\") pod \"payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s\" (UID: \"2778fdc5-5b89-4783-856c-4e26b7fe30f6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s" Nov 03 22:39:55.180762 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:55.180717 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26" Nov 03 22:39:55.259655 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:55.259610 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2778fdc5-5b89-4783-856c-4e26b7fe30f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s\" (UID: \"2778fdc5-5b89-4783-856c-4e26b7fe30f6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s" Nov 03 22:39:55.259881 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:55.259747 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjqmx\" (UniqueName: \"kubernetes.io/projected/2778fdc5-5b89-4783-856c-4e26b7fe30f6-kube-api-access-hjqmx\") pod \"payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s\" (UID: \"2778fdc5-5b89-4783-856c-4e26b7fe30f6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s" Nov 03 22:39:55.261618 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:55.261558 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2778fdc5-5b89-4783-856c-4e26b7fe30f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s\" (UID: \"2778fdc5-5b89-4783-856c-4e26b7fe30f6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s" Nov 03 22:39:55.319584 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:55.319421 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjqmx\" (UniqueName: \"kubernetes.io/projected/2778fdc5-5b89-4783-856c-4e26b7fe30f6-kube-api-access-hjqmx\") pod \"payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s\" (UID: \"2778fdc5-5b89-4783-856c-4e26b7fe30f6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s" Nov 03 22:39:55.582546 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:55.582503 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26"] Nov 03 22:39:55.615456 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:55.615425 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s" Nov 03 22:39:56.009201 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:56.009167 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s"] Nov 03 22:39:56.010105 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:39:56.010037 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2778fdc5_5b89_4783_856c_4e26b7fe30f6.slice/crio-35734b03d55b9c84912121a75e8dd479b9fc0741c5e2f35efa98ef386dfe6ca1 WatchSource:0}: Error finding container 35734b03d55b9c84912121a75e8dd479b9fc0741c5e2f35efa98ef386dfe6ca1: Status 404 returned error can't find the container with id 35734b03d55b9c84912121a75e8dd479b9fc0741c5e2f35efa98ef386dfe6ca1 Nov 03 22:39:56.078717 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:56.078676 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s" event={"ID":"2778fdc5-5b89-4783-856c-4e26b7fe30f6","Type":"ContainerStarted","Data":"35734b03d55b9c84912121a75e8dd479b9fc0741c5e2f35efa98ef386dfe6ca1"} Nov 03 22:39:56.081045 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:56.080989 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26" event={"ID":"6a069ee1-c643-4a4a-8632-6e02db66d490","Type":"ContainerStarted","Data":"a9667bd04468a9cd0a8804a35170e0efd8def1bbbcb2800cff2d572d5bef5b82"} Nov 03 22:39:56.081263 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:56.081106 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26" event={"ID":"6a069ee1-c643-4a4a-8632-6e02db66d490","Type":"ContainerStarted","Data":"748e060ef6e0acf66727cc75e0f2d3f1bac67996981e436b7b75de836e541545"} Nov 03 22:39:56.081263 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:56.081145 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26" Nov 03 22:39:56.202120 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:56.202036 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26" podStartSLOduration=2.202014489 podStartE2EDuration="2.202014489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:56.196603518 +0000 UTC m=+1963.790635493" watchObservedRunningTime="2026-11-03 22:39:56.202014489 +0000 UTC m=+1963.796046464" Nov 03 22:39:57.093217 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.093175 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s" event={"ID":"2778fdc5-5b89-4783-856c-4e26b7fe30f6","Type":"ContainerStarted","Data":"18b75e2a87c801836d773d3293e3f817efd885932fcfddc0effca79777104681"} Nov 03 22:39:57.093754 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.093248 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s" Nov 03 22:39:57.212700 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.212644 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s" podStartSLOduration=3.212629132 podStartE2EDuration="3.212629132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:57.209409557 +0000 UTC m=+1964.803441531" watchObservedRunningTime="2026-11-03 22:39:57.212629132 +0000 UTC m=+1964.806661106" Nov 03 22:39:57.264822 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.264782 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh"] Nov 03 22:39:57.270015 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.269987 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:39:57.291671 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.291625 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh\" (UID: \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:39:57.291873 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.291690 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh\" (UID: \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:39:57.291873 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.291772 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh\" (UID: \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:39:57.292028 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.292000 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87rvk\" (UniqueName: \"kubernetes.io/projected/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-kube-api-access-87rvk\") pod \"maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh\" (UID: \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:39:57.318852 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.318811 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh"] Nov 03 22:39:57.409191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.407977 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-094053-serving-cert\"" Nov 03 22:39:57.419908 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.417444 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh\" (UID: \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:39:57.419908 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.417515 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh\" (UID: \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:39:57.419908 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.417551 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh\" (UID: \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:39:57.419908 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.417689 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87rvk\" (UniqueName: \"kubernetes.io/projected/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-kube-api-access-87rvk\") pod \"maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh\" (UID: \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:39:57.419908 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:39:57.418167 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-serving-cert: secret "maas-api-e2e-shared-a-w0-094053-serving-cert" not found Nov 03 22:39:57.419908 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:39:57.418274 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-maas-api-tls podName:1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6 nodeName:}" failed. No retries permitted until 2026-11-03 22:39:57.918235229 +0000 UTC m=+1965.512267184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-maas-api-tls") pod "maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" (UID: "1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6") : secret "maas-api-e2e-shared-a-w0-094053-serving-cert" not found Nov 03 22:39:57.419908 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.419123 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh\" (UID: \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:39:57.432813 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.432766 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh\" (UID: \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:39:57.543093 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.542797 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87rvk\" (UniqueName: \"kubernetes.io/projected/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-kube-api-access-87rvk\") pod \"maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh\" (UID: \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:39:57.957879 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.957831 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh\" (UID: \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:39:57.962023 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:57.961981 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh\" (UID: \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:39:58.187617 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:58.187575 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:39:58.579290 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:39:58.579253 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8a5e32_bb39_43c1_86ab_fdbb6a5f7dd6.slice/crio-8bfb1a97b6c4145a5b4f8a2d324ff060c0c923b845b83403fdd3a9bd3d0873fd WatchSource:0}: Error finding container 8bfb1a97b6c4145a5b4f8a2d324ff060c0c923b845b83403fdd3a9bd3d0873fd: Status 404 returned error can't find the container with id 8bfb1a97b6c4145a5b4f8a2d324ff060c0c923b845b83403fdd3a9bd3d0873fd Nov 03 22:39:58.582042 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:58.582003 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh"] Nov 03 22:39:59.111677 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:59.111638 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" event={"ID":"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6","Type":"ContainerStarted","Data":"8bfb1a97b6c4145a5b4f8a2d324ff060c0c923b845b83403fdd3a9bd3d0873fd"} Nov 03 22:39:59.640433 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:59.640389 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd"] Nov 03 22:39:59.661170 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:59.653265 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd" Nov 03 22:39:59.676085 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:59.667808 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd"] Nov 03 22:39:59.723143 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:59.716015 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j678f\" (UniqueName: \"kubernetes.io/projected/d8b839e0-50e4-43d1-8b8d-7f62b553f3a8-kube-api-access-j678f\") pod \"payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd\" (UID: \"d8b839e0-50e4-43d1-8b8d-7f62b553f3a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd" Nov 03 22:39:59.744937 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:59.744896 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b839e0-50e4-43d1-8b8d-7f62b553f3a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd\" (UID: \"d8b839e0-50e4-43d1-8b8d-7f62b553f3a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd" Nov 03 22:39:59.821101 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:59.743801 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-bc88eb47\"" Nov 03 22:39:59.916996 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:59.743711 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-bc88eb47-dockercfg-jxfmd\"" Nov 03 22:39:59.984297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:59.953487 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b839e0-50e4-43d1-8b8d-7f62b553f3a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd\" (UID: \"d8b839e0-50e4-43d1-8b8d-7f62b553f3a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd" Nov 03 22:39:59.984297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:59.953620 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j678f\" (UniqueName: \"kubernetes.io/projected/d8b839e0-50e4-43d1-8b8d-7f62b553f3a8-kube-api-access-j678f\") pod \"payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd\" (UID: \"d8b839e0-50e4-43d1-8b8d-7f62b553f3a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd" Nov 03 22:39:59.984297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:39:59.955164 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b839e0-50e4-43d1-8b8d-7f62b553f3a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd\" (UID: \"d8b839e0-50e4-43d1-8b8d-7f62b553f3a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd" Nov 03 22:40:00.023534 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:00.022543 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6" Nov 03 22:40:00.025901 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:00.025870 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" Nov 03 22:40:00.033189 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:00.033157 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j678f\" (UniqueName: \"kubernetes.io/projected/d8b839e0-50e4-43d1-8b8d-7f62b553f3a8-kube-api-access-j678f\") pod \"payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd\" (UID: \"d8b839e0-50e4-43d1-8b8d-7f62b553f3a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd" Nov 03 22:40:00.123054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:00.123010 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" event={"ID":"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6","Type":"ContainerStarted","Data":"a3597d06ddfdb8ea25fc29fc40ff4be1908f8acc8abda143ea6455d66a450220"} Nov 03 22:40:00.123239 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:00.123133 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:40:00.281712 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:00.280427 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" podStartSLOduration=2.438887089 podStartE2EDuration="3.280408853s" totalImagesPullingTime="841.521764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:00.266793642 +0000 UTC m=+1967.860825615" watchObservedRunningTime="2026-11-03 22:40:00.280408853 +0000 UTC m=+1967.874440826" Nov 03 22:40:00.322329 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:00.322289 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd" Nov 03 22:40:00.774776 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:40:00.774734 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b839e0_50e4_43d1_8b8d_7f62b553f3a8.slice/crio-3a2a9e2ba54199ab45f1cd68b49c6d500ce252023c1a5f2d4372b88546e9d780 WatchSource:0}: Error finding container 3a2a9e2ba54199ab45f1cd68b49c6d500ce252023c1a5f2d4372b88546e9d780: Status 404 returned error can't find the container with id 3a2a9e2ba54199ab45f1cd68b49c6d500ce252023c1a5f2d4372b88546e9d780 Nov 03 22:40:00.775239 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:00.775117 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd"] Nov 03 22:40:01.133261 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:01.133157 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd" event={"ID":"d8b839e0-50e4-43d1-8b8d-7f62b553f3a8","Type":"ContainerStarted","Data":"1ba45996d883c63985faff339b3e5f38e9c69133dc30d3df73c0d029a3bc6125"} Nov 03 22:40:01.133261 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:01.133217 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd" event={"ID":"d8b839e0-50e4-43d1-8b8d-7f62b553f3a8","Type":"ContainerStarted","Data":"3a2a9e2ba54199ab45f1cd68b49c6d500ce252023c1a5f2d4372b88546e9d780"} Nov 03 22:40:01.133261 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:01.133256 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd" Nov 03 22:40:01.253251 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:01.253187 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd" podStartSLOduration=2.253171076 podStartE2EDuration="2.253171076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:01.250037229 +0000 UTC m=+1968.844069204" watchObservedRunningTime="2026-11-03 22:40:01.253171076 +0000 UTC m=+1968.847203049" Nov 03 22:40:06.206084 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:06.205267 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:40:07.097056 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:07.097022 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26" Nov 03 22:40:09.116190 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:09.116154 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s" Nov 03 22:40:12.026475 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:40:12.026434 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:40:12.146873 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:12.146841 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd" Nov 03 22:40:20.769808 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:20.765805 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv"] Nov 03 22:40:20.882020 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:20.881970 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922"] Nov 03 22:40:25.767247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:25.767193 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv" podUID="7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7" containerName="payload-processing" containerID="cri-o://2a76f83baf1a5d8543af6c2cfab5190d8e217c7cf46b75efc5cde5b50435916e" gracePeriod=25 Nov 03 22:40:25.883154 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:25.883102 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922" podUID="1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5" containerName="payload-pre-processing" containerID="cri-o://f858dfd92eb55ef8d24080f33062975b4450820dacc2db1a23870be12cfafc7a" gracePeriod=25 Nov 03 22:40:26.088596 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.088560 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv" Nov 03 22:40:26.143104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.143049 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7-plugins-config-volume\") pod \"7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7\" (UID: \"7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7\") " Nov 03 22:40:26.143394 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.143372 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7-kube-api-access-b9qd5\" (UniqueName: \"kubernetes.io/projected/7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7-kube-api-access-b9qd5\") pod \"7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7\" (UID: \"7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7\") " Nov 03 22:40:26.143628 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.143601 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7-plugins-config-volume" pod "7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7" (UID: "7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:40:26.144265 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.144236 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:26.146274 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.146242 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7-kube-api-access-b9qd5" pod "7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7" (UID: "7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7"). InnerVolumeSpecName "kube-api-access-b9qd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:26.196355 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.196321 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922" Nov 03 22:40:26.245095 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.245018 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5-kube-api-access-7mrqw\" (UniqueName: \"kubernetes.io/projected/1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5-kube-api-access-7mrqw\") pod \"1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5\" (UID: \"1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5\") " Nov 03 22:40:26.245432 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.245406 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5-plugins-config-volume\") pod \"1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5\" (UID: \"1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5\") " Nov 03 22:40:26.245806 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.245780 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5-plugins-config-volume" pod "1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5" (UID: "1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:40:26.246047 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.246024 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:26.246126 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.246055 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9qd5\" (UniqueName: \"kubernetes.io/projected/7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7-kube-api-access-b9qd5\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:26.247399 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.247371 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5-kube-api-access-7mrqw" pod "1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5" (UID: "1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5"). InnerVolumeSpecName "kube-api-access-7mrqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:26.348103 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.347982 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mrqw\" (UniqueName: \"kubernetes.io/projected/1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5-kube-api-access-7mrqw\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:26.481426 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.481381 5084 generic.go:356] "Generic (PLEG): container finished" podID="1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5" containerID="f858dfd92eb55ef8d24080f33062975b4450820dacc2db1a23870be12cfafc7a" exitCode=0 Nov 03 22:40:26.481638 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.481432 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922" Nov 03 22:40:26.481638 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.481441 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922" event={"ID":"1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5","Type":"ContainerDied","Data":"f858dfd92eb55ef8d24080f33062975b4450820dacc2db1a23870be12cfafc7a"} Nov 03 22:40:26.481638 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.481537 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922" event={"ID":"1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5","Type":"ContainerDied","Data":"e603fe3105ffdf25935c23d9541419b95262a28daeb6a634512cbe9e3e755a53"} Nov 03 22:40:26.481638 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.481569 5084 scope.go:122] "RemoveContainer" containerID="f858dfd92eb55ef8d24080f33062975b4450820dacc2db1a23870be12cfafc7a" Nov 03 22:40:26.483479 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.483449 5084 generic.go:356] "Generic (PLEG): container finished" podID="7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7" containerID="2a76f83baf1a5d8543af6c2cfab5190d8e217c7cf46b75efc5cde5b50435916e" exitCode=0 Nov 03 22:40:26.483642 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.483622 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv" Nov 03 22:40:26.483730 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.483664 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv" event={"ID":"7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7","Type":"ContainerDied","Data":"2a76f83baf1a5d8543af6c2cfab5190d8e217c7cf46b75efc5cde5b50435916e"} Nov 03 22:40:26.483730 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.483718 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv" event={"ID":"7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7","Type":"ContainerDied","Data":"2a394dcabb6963210c4dd3a23ad2c92a7ba99b2362c332d8cb84c6e5f27f97bd"} Nov 03 22:40:26.495957 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.495927 5084 scope.go:122] "RemoveContainer" containerID="f858dfd92eb55ef8d24080f33062975b4450820dacc2db1a23870be12cfafc7a" Nov 03 22:40:26.496451 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:40:26.496424 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f858dfd92eb55ef8d24080f33062975b4450820dacc2db1a23870be12cfafc7a\": container with ID starting with f858dfd92eb55ef8d24080f33062975b4450820dacc2db1a23870be12cfafc7a not found: ID does not exist" containerID="f858dfd92eb55ef8d24080f33062975b4450820dacc2db1a23870be12cfafc7a" Nov 03 22:40:26.496563 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.496463 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f858dfd92eb55ef8d24080f33062975b4450820dacc2db1a23870be12cfafc7a"} err="failed to get container status \"f858dfd92eb55ef8d24080f33062975b4450820dacc2db1a23870be12cfafc7a\": rpc error: code = NotFound desc = could not find container \"f858dfd92eb55ef8d24080f33062975b4450820dacc2db1a23870be12cfafc7a\": container with ID starting with f858dfd92eb55ef8d24080f33062975b4450820dacc2db1a23870be12cfafc7a not found: ID does not exist" Nov 03 22:40:26.496563 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.496490 5084 scope.go:122] "RemoveContainer" containerID="2a76f83baf1a5d8543af6c2cfab5190d8e217c7cf46b75efc5cde5b50435916e" Nov 03 22:40:26.512378 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.512262 5084 scope.go:122] "RemoveContainer" containerID="2a76f83baf1a5d8543af6c2cfab5190d8e217c7cf46b75efc5cde5b50435916e" Nov 03 22:40:26.515103 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:40:26.515059 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a76f83baf1a5d8543af6c2cfab5190d8e217c7cf46b75efc5cde5b50435916e\": container with ID starting with 2a76f83baf1a5d8543af6c2cfab5190d8e217c7cf46b75efc5cde5b50435916e not found: ID does not exist" containerID="2a76f83baf1a5d8543af6c2cfab5190d8e217c7cf46b75efc5cde5b50435916e" Nov 03 22:40:26.515240 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.515114 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a76f83baf1a5d8543af6c2cfab5190d8e217c7cf46b75efc5cde5b50435916e"} err="failed to get container status \"2a76f83baf1a5d8543af6c2cfab5190d8e217c7cf46b75efc5cde5b50435916e\": rpc error: code = NotFound desc = could not find container \"2a76f83baf1a5d8543af6c2cfab5190d8e217c7cf46b75efc5cde5b50435916e\": container with ID starting with 2a76f83baf1a5d8543af6c2cfab5190d8e217c7cf46b75efc5cde5b50435916e not found: ID does not exist" Nov 03 22:40:26.650456 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.650367 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv"] Nov 03 22:40:26.654611 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.654576 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-37b70b-6bdbb9d8fd-88rsv"] Nov 03 22:40:26.766167 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.766122 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922"] Nov 03 22:40:26.772540 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:26.772499 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-37b70b-8db488b6c-wf922"] Nov 03 22:40:27.764169 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:27.764129 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r"] Nov 03 22:40:27.764421 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:27.764392 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" podUID="0c395459-9f53-4792-9b3a-7a3a1f733a65" containerName="istio-proxy" containerID="cri-o://0419033b12c66dd1b10445fe72a2da15fbfc2cd65dad44bc23f86000ff746db2" gracePeriod=30 Nov 03 22:40:28.629919 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:28.629882 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5" path="/var/lib/kubelet/pods/1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5/volumes" Nov 03 22:40:28.630393 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:28.630378 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7" path="/var/lib/kubelet/pods/7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7/volumes" Nov 03 22:40:29.014122 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:29.014011 5084 patch_prober.go:29] interesting pod/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:40:29.014122 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:29.014089 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" podUID="0c395459-9f53-4792-9b3a-7a3a1f733a65" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:40:31.341424 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.341383 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n"] Nov 03 22:40:31.342193 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.342172 5084 cpu_manager.go:414] "containerMap: removing container" podUID="7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7" containerName="payload-processing" Nov 03 22:40:31.342274 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.342196 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7" containerName="payload-processing" Nov 03 22:40:31.342274 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.342216 5084 cpu_manager.go:414] "containerMap: removing container" podUID="1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5" containerName="payload-pre-processing" Nov 03 22:40:31.342274 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.342223 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5" containerName="payload-pre-processing" Nov 03 22:40:31.342410 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.342396 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aea9fcd-0d68-4fbb-9f7f-05972a33dfc7" containerName="payload-processing" Nov 03 22:40:31.342450 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.342415 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d8a3d90-bd3f-453d-b244-9a7a0f21bbc5" containerName="payload-pre-processing" Nov 03 22:40:31.347556 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.347527 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:40:31.399058 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.399017 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n"] Nov 03 22:40:31.401489 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.401458 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a9ef23-serving-cert\"" Nov 03 22:40:31.452419 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.452374 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-metrics-certs\") pod \"maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n\" (UID: \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:40:31.452708 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.452682 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-openshift-service-ca\") pod \"maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n\" (UID: \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:40:31.452794 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.452752 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqvw9\" (UniqueName: \"kubernetes.io/projected/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-kube-api-access-wqvw9\") pod \"maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n\" (UID: \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:40:31.452894 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.452868 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-maas-api-tls\") pod \"maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n\" (UID: \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:40:31.553818 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.553771 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-openshift-service-ca\") pod \"maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n\" (UID: \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:40:31.554012 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.553858 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqvw9\" (UniqueName: \"kubernetes.io/projected/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-kube-api-access-wqvw9\") pod \"maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n\" (UID: \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:40:31.554012 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.553893 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-maas-api-tls\") pod \"maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n\" (UID: \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:40:31.554162 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.554030 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-metrics-certs\") pod \"maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n\" (UID: \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:40:31.554162 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:40:31.554094 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-serving-cert: secret "maas-api-e2e-api-b-a9ef23-serving-cert" not found Nov 03 22:40:31.554278 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:40:31.554183 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-maas-api-tls podName:0da90ec4-3e54-4bfc-984b-f3d60baa79ea nodeName:}" failed. No retries permitted until 2026-11-03 22:40:32.054165372 +0000 UTC m=+1999.648197328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-maas-api-tls") pod "maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" (UID: "0da90ec4-3e54-4bfc-984b-f3d60baa79ea") : secret "maas-api-e2e-api-b-a9ef23-serving-cert" not found Nov 03 22:40:31.554679 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.554656 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-openshift-service-ca\") pod \"maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n\" (UID: \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:40:31.556767 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.556738 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-metrics-certs\") pod \"maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n\" (UID: \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:40:31.646920 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:31.640846 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqvw9\" (UniqueName: \"kubernetes.io/projected/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-kube-api-access-wqvw9\") pod \"maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n\" (UID: \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:40:32.067979 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:32.067935 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-maas-api-tls\") pod \"maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n\" (UID: \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:40:32.070551 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:32.070523 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-maas-api-tls\") pod \"maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n\" (UID: \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:40:32.265411 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:32.265372 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:40:32.655586 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:32.655551 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n"] Nov 03 22:40:32.657058 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:40:32.657026 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da90ec4_3e54_4bfc_984b_f3d60baa79ea.slice/crio-6243714042f12443f113025713a21980511704942cdd42f36684f5b7cd5676ac WatchSource:0}: Error finding container 6243714042f12443f113025713a21980511704942cdd42f36684f5b7cd5676ac: Status 404 returned error can't find the container with id 6243714042f12443f113025713a21980511704942cdd42f36684f5b7cd5676ac Nov 03 22:40:33.085709 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.085676 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:40:33.186567 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.186521 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c395459-9f53-4792-9b3a-7a3a1f733a65-kube-api-access-9zsm7\" (UniqueName: \"kubernetes.io/projected/0c395459-9f53-4792-9b3a-7a3a1f733a65-kube-api-access-9zsm7\") pod \"0c395459-9f53-4792-9b3a-7a3a1f733a65\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " Nov 03 22:40:33.186773 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.186621 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-podinfo\") pod \"0c395459-9f53-4792-9b3a-7a3a1f733a65\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " Nov 03 22:40:33.186773 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.186698 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-data\") pod \"0c395459-9f53-4792-9b3a-7a3a1f733a65\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " Nov 03 22:40:33.186907 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.186815 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-credential-socket\") pod \"0c395459-9f53-4792-9b3a-7a3a1f733a65\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " Nov 03 22:40:33.186907 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.186850 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-token\" (UniqueName: \"kubernetes.io/projected/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-token\") pod \"0c395459-9f53-4792-9b3a-7a3a1f733a65\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " Nov 03 22:40:33.186907 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.186887 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-envoy\") pod \"0c395459-9f53-4792-9b3a-7a3a1f733a65\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " Nov 03 22:40:33.186907 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.186941 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-workload-socket\") pod \"0c395459-9f53-4792-9b3a-7a3a1f733a65\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " Nov 03 22:40:33.187202 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.187012 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-workload-certs\") pod \"0c395459-9f53-4792-9b3a-7a3a1f733a65\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " Nov 03 22:40:33.187202 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.187051 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c395459-9f53-4792-9b3a-7a3a1f733a65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c395459-9f53-4792-9b3a-7a3a1f733a65-istiod-ca-cert\") pod \"0c395459-9f53-4792-9b3a-7a3a1f733a65\" (UID: \"0c395459-9f53-4792-9b3a-7a3a1f733a65\") " Nov 03 22:40:33.187202 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.187120 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-data" pod "0c395459-9f53-4792-9b3a-7a3a1f733a65" (UID: "0c395459-9f53-4792-9b3a-7a3a1f733a65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:33.187202 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.187155 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-credential-socket" pod "0c395459-9f53-4792-9b3a-7a3a1f733a65" (UID: "0c395459-9f53-4792-9b3a-7a3a1f733a65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:33.187412 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.187367 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-workload-socket" pod "0c395459-9f53-4792-9b3a-7a3a1f733a65" (UID: "0c395459-9f53-4792-9b3a-7a3a1f733a65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:33.187705 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.187663 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c395459-9f53-4792-9b3a-7a3a1f733a65-istiod-ca-cert" pod "0c395459-9f53-4792-9b3a-7a3a1f733a65" (UID: "0c395459-9f53-4792-9b3a-7a3a1f733a65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:40:33.187882 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.187849 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-workload-certs" pod "0c395459-9f53-4792-9b3a-7a3a1f733a65" (UID: "0c395459-9f53-4792-9b3a-7a3a1f733a65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:33.187982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.187933 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:33.187982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.187950 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c395459-9f53-4792-9b3a-7a3a1f733a65-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:33.187982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.187964 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:33.187982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.187977 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:33.189563 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.189527 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-podinfo" pod "0c395459-9f53-4792-9b3a-7a3a1f733a65" (UID: "0c395459-9f53-4792-9b3a-7a3a1f733a65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:40:33.189675 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.189633 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c395459-9f53-4792-9b3a-7a3a1f733a65-kube-api-access-9zsm7" pod "0c395459-9f53-4792-9b3a-7a3a1f733a65" (UID: "0c395459-9f53-4792-9b3a-7a3a1f733a65"). InnerVolumeSpecName "kube-api-access-9zsm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:33.189675 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.189647 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-envoy" pod "0c395459-9f53-4792-9b3a-7a3a1f733a65" (UID: "0c395459-9f53-4792-9b3a-7a3a1f733a65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:33.189777 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.189677 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-token" pod "0c395459-9f53-4792-9b3a-7a3a1f733a65" (UID: "0c395459-9f53-4792-9b3a-7a3a1f733a65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:33.289683 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.289639 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:33.289683 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.289684 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:33.289972 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.289698 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c395459-9f53-4792-9b3a-7a3a1f733a65-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:33.289972 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.289711 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zsm7\" (UniqueName: \"kubernetes.io/projected/0c395459-9f53-4792-9b3a-7a3a1f733a65-kube-api-access-9zsm7\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:33.289972 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.289726 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c395459-9f53-4792-9b3a-7a3a1f733a65-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:40:33.546349 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.546321 5084 generic.go:356] "Generic (PLEG): container finished" podID="0c395459-9f53-4792-9b3a-7a3a1f733a65" containerID="0419033b12c66dd1b10445fe72a2da15fbfc2cd65dad44bc23f86000ff746db2" exitCode=0 Nov 03 22:40:33.546436 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.546372 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" Nov 03 22:40:33.546490 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.546416 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" event={"ID":"0c395459-9f53-4792-9b3a-7a3a1f733a65","Type":"ContainerDied","Data":"0419033b12c66dd1b10445fe72a2da15fbfc2cd65dad44bc23f86000ff746db2"} Nov 03 22:40:33.546558 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.546488 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r" event={"ID":"0c395459-9f53-4792-9b3a-7a3a1f733a65","Type":"ContainerDied","Data":"7aafe600b2f00fb5952814f60a9c6863834cd5b9f09a0cc5e05f73a459e49712"} Nov 03 22:40:33.546558 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.546523 5084 scope.go:122] "RemoveContainer" containerID="0419033b12c66dd1b10445fe72a2da15fbfc2cd65dad44bc23f86000ff746db2" Nov 03 22:40:33.548323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.548294 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" event={"ID":"0da90ec4-3e54-4bfc-984b-f3d60baa79ea","Type":"ContainerStarted","Data":"6243714042f12443f113025713a21980511704942cdd42f36684f5b7cd5676ac"} Nov 03 22:40:33.569805 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.569781 5084 scope.go:122] "RemoveContainer" containerID="0419033b12c66dd1b10445fe72a2da15fbfc2cd65dad44bc23f86000ff746db2" Nov 03 22:40:33.570301 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:40:33.570268 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0419033b12c66dd1b10445fe72a2da15fbfc2cd65dad44bc23f86000ff746db2\": container with ID starting with 0419033b12c66dd1b10445fe72a2da15fbfc2cd65dad44bc23f86000ff746db2 not found: ID does not exist" containerID="0419033b12c66dd1b10445fe72a2da15fbfc2cd65dad44bc23f86000ff746db2" Nov 03 22:40:33.570417 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.570376 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0419033b12c66dd1b10445fe72a2da15fbfc2cd65dad44bc23f86000ff746db2"} err="failed to get container status \"0419033b12c66dd1b10445fe72a2da15fbfc2cd65dad44bc23f86000ff746db2\": rpc error: code = NotFound desc = could not find container \"0419033b12c66dd1b10445fe72a2da15fbfc2cd65dad44bc23f86000ff746db2\": container with ID starting with 0419033b12c66dd1b10445fe72a2da15fbfc2cd65dad44bc23f86000ff746db2 not found: ID does not exist" Nov 03 22:40:33.722779 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.722724 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r"] Nov 03 22:40:33.736521 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:33.730803 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f7e6db-openshift-default-9dcdbb8f7-mn47r"] Nov 03 22:40:34.561244 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:34.561206 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" event={"ID":"0da90ec4-3e54-4bfc-984b-f3d60baa79ea","Type":"ContainerStarted","Data":"1566de9592be6f7a313bc22b5db3c7430b1d3a16f731b933bfdab41dd15d3177"} Nov 03 22:40:34.561454 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:34.561314 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:40:34.629396 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:34.629360 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c395459-9f53-4792-9b3a-7a3a1f733a65" path="/var/lib/kubelet/pods/0c395459-9f53-4792-9b3a-7a3a1f733a65/volumes" Nov 03 22:40:34.677618 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:34.677553 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" podStartSLOduration=2.80931203 podStartE2EDuration="3.677532985s" totalImagesPullingTime="868.220955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:34.676325967 +0000 UTC m=+2002.270357942" watchObservedRunningTime="2026-11-03 22:40:34.677532985 +0000 UTC m=+2002.271564959" Nov 03 22:40:40.576059 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:40.576026 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:40:42.025157 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:40:42.025115 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:40:54.151644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.151556 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6"] Nov 03 22:40:54.152378 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.152358 5084 cpu_manager.go:414] "containerMap: removing container" podUID="0c395459-9f53-4792-9b3a-7a3a1f733a65" containerName="istio-proxy" Nov 03 22:40:54.152431 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.152379 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c395459-9f53-4792-9b3a-7a3a1f733a65" containerName="istio-proxy" Nov 03 22:40:54.152535 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.152525 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c395459-9f53-4792-9b3a-7a3a1f733a65" containerName="istio-proxy" Nov 03 22:40:54.157843 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.157745 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:40:54.210209 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.210168 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6"] Nov 03 22:40:54.255843 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.255811 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-345da9-serving-cert\"" Nov 03 22:40:54.306709 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.306665 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6\" (UID: \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:40:54.306709 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.306712 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-metrics-certs\") pod \"maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6\" (UID: \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:40:54.307028 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.307002 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-maas-api-tls\") pod \"maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6\" (UID: \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:40:54.307208 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.307187 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bchkv\" (UniqueName: \"kubernetes.io/projected/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-kube-api-access-bchkv\") pod \"maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6\" (UID: \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:40:54.408835 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.408731 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6\" (UID: \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:40:54.408835 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.408776 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-metrics-certs\") pod \"maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6\" (UID: \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:40:54.409145 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.408935 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-maas-api-tls\") pod \"maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6\" (UID: \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:40:54.409145 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:40:54.409103 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-serving-cert: secret "maas-api-e2e-inf-a-345da9-serving-cert" not found Nov 03 22:40:54.409284 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.409151 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bchkv\" (UniqueName: \"kubernetes.io/projected/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-kube-api-access-bchkv\") pod \"maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6\" (UID: \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:40:54.409284 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:40:54.409182 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-maas-api-tls podName:b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2 nodeName:}" failed. No retries permitted until 2026-11-03 22:40:54.90915932 +0000 UTC m=+2022.503191288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-maas-api-tls") pod "maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" (UID: "b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2") : secret "maas-api-e2e-inf-a-345da9-serving-cert" not found Nov 03 22:40:54.409634 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.409612 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6\" (UID: \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:40:54.411541 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.411515 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-metrics-certs\") pod \"maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6\" (UID: \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:40:54.486098 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.480639 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bchkv\" (UniqueName: \"kubernetes.io/projected/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-kube-api-access-bchkv\") pod \"maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6\" (UID: \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:40:54.946245 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.946202 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-maas-api-tls\") pod \"maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6\" (UID: \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:40:54.949040 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:54.949011 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-maas-api-tls\") pod \"maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6\" (UID: \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:40:55.076081 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:55.076023 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:40:55.607235 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:55.607197 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6"] Nov 03 22:40:55.608477 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:40:55.608442 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cc5504_a4f0_4f55_91bd_e2bc277fc7a2.slice/crio-05976a1c85c5d85c8f45e45aae7bb106f264cfc344be19d012e716519f569457 WatchSource:0}: Error finding container 05976a1c85c5d85c8f45e45aae7bb106f264cfc344be19d012e716519f569457: Status 404 returned error can't find the container with id 05976a1c85c5d85c8f45e45aae7bb106f264cfc344be19d012e716519f569457 Nov 03 22:40:55.741150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:55.741108 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" event={"ID":"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2","Type":"ContainerStarted","Data":"05976a1c85c5d85c8f45e45aae7bb106f264cfc344be19d012e716519f569457"} Nov 03 22:40:56.752590 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:56.752547 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" event={"ID":"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2","Type":"ContainerStarted","Data":"423be09eb177a2e6b84c416ae20b6f5e1e1087848c81d251740cb50bf6bdd702"} Nov 03 22:40:56.753144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:56.752801 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:40:56.907864 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:40:56.907735 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" podStartSLOduration=1.8816371950000002 podStartE2EDuration="2.907713768s" totalImagesPullingTime="1.026076573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:56.90326954 +0000 UTC m=+2024.497301514" watchObservedRunningTime="2026-11-03 22:40:56.907713768 +0000 UTC m=+2024.501745743" Nov 03 22:41:02.768530 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:02.768494 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:41:04.200847 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:04.200808 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9"] Nov 03 22:41:04.206133 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:04.206100 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:41:04.228243 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:04.228210 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31886473-ca58-4b81-8895-477545b480e6-openshift-service-ca\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9\" (UID: \"31886473-ca58-4b81-8895-477545b480e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:41:04.228400 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:04.228323 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n96q7\" (UniqueName: \"kubernetes.io/projected/31886473-ca58-4b81-8895-477545b480e6-kube-api-access-n96q7\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9\" (UID: \"31886473-ca58-4b81-8895-477545b480e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:41:04.228470 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:04.228407 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31886473-ca58-4b81-8895-477545b480e6-metrics-certs\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9\" (UID: \"31886473-ca58-4b81-8895-477545b480e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:41:04.228509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:04.228488 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31886473-ca58-4b81-8895-477545b480e6-maas-api-tls\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9\" (UID: \"31886473-ca58-4b81-8895-477545b480e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:41:04.261666 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:04.261593 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9"] Nov 03 22:41:04.263397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:04.263359 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-bc88eb47-serving-cert\"" Nov 03 22:41:04.329535 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:04.329486 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31886473-ca58-4b81-8895-477545b480e6-metrics-certs\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9\" (UID: \"31886473-ca58-4b81-8895-477545b480e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:41:04.329743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:04.329565 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31886473-ca58-4b81-8895-477545b480e6-maas-api-tls\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9\" (UID: \"31886473-ca58-4b81-8895-477545b480e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:41:04.329743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:04.329649 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31886473-ca58-4b81-8895-477545b480e6-openshift-service-ca\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9\" (UID: \"31886473-ca58-4b81-8895-477545b480e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:41:04.329743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:04.329713 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n96q7\" (UniqueName: \"kubernetes.io/projected/31886473-ca58-4b81-8895-477545b480e6-kube-api-access-n96q7\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9\" (UID: \"31886473-ca58-4b81-8895-477545b480e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:41:04.329863 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:41:04.329724 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-serving-cert: secret "maas-api-e2e-mt-bc88eb47-serving-cert" not found Nov 03 22:41:04.329863 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:41:04.329820 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31886473-ca58-4b81-8895-477545b480e6-maas-api-tls podName:31886473-ca58-4b81-8895-477545b480e6 nodeName:}" failed. No retries permitted until 2026-11-03 22:41:04.829799706 +0000 UTC m=+2032.423831671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/31886473-ca58-4b81-8895-477545b480e6-maas-api-tls") pod "maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" (UID: "31886473-ca58-4b81-8895-477545b480e6") : secret "maas-api-e2e-mt-bc88eb47-serving-cert" not found Nov 03 22:41:04.330446 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:04.330422 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31886473-ca58-4b81-8895-477545b480e6-openshift-service-ca\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9\" (UID: \"31886473-ca58-4b81-8895-477545b480e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:41:04.332189 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:04.332165 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31886473-ca58-4b81-8895-477545b480e6-metrics-certs\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9\" (UID: \"31886473-ca58-4b81-8895-477545b480e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:41:04.408574 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:04.408535 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n96q7\" (UniqueName: \"kubernetes.io/projected/31886473-ca58-4b81-8895-477545b480e6-kube-api-access-n96q7\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9\" (UID: \"31886473-ca58-4b81-8895-477545b480e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:41:04.847079 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:04.847017 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31886473-ca58-4b81-8895-477545b480e6-maas-api-tls\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9\" (UID: \"31886473-ca58-4b81-8895-477545b480e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:41:04.850117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:04.850082 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31886473-ca58-4b81-8895-477545b480e6-maas-api-tls\") pod \"maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9\" (UID: \"31886473-ca58-4b81-8895-477545b480e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:41:05.123460 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:05.123373 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:41:05.514472 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:05.514436 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9"] Nov 03 22:41:05.515195 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:41:05.515165 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31886473_ca58_4b81_8895_477545b480e6.slice/crio-12a390ab7fdec40294070bf25a7ad55e5b045fd263b51caec451d444e037daec WatchSource:0}: Error finding container 12a390ab7fdec40294070bf25a7ad55e5b045fd263b51caec451d444e037daec: Status 404 returned error can't find the container with id 12a390ab7fdec40294070bf25a7ad55e5b045fd263b51caec451d444e037daec Nov 03 22:41:05.516967 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:05.516949 5084 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:41:05.882183 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:05.882140 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" event={"ID":"31886473-ca58-4b81-8895-477545b480e6","Type":"ContainerStarted","Data":"12a390ab7fdec40294070bf25a7ad55e5b045fd263b51caec451d444e037daec"} Nov 03 22:41:06.897285 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:06.897239 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" event={"ID":"31886473-ca58-4b81-8895-477545b480e6","Type":"ContainerStarted","Data":"7144e629bf0f765bb8b0989e276fc3029de9a16390e5066907cf2da555b1f60f"} Nov 03 22:41:06.898009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:06.897323 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:41:07.018559 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:07.018493 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" podStartSLOduration=2.103109379 podStartE2EDuration="3.018473274s" totalImagesPullingTime="915.363895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:07.014479192 +0000 UTC m=+2034.608511168" watchObservedRunningTime="2026-11-03 22:41:07.018473274 +0000 UTC m=+2034.612505241" Nov 03 22:41:09.351738 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:09.351691 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46"] Nov 03 22:41:09.359076 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:09.359036 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:41:09.412280 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:09.412245 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-2a6801-serving-cert\"" Nov 03 22:41:09.430289 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:09.430246 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/060aaaff-87eb-4800-99af-122329708c89-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46\" (UID: \"060aaaff-87eb-4800-99af-122329708c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:41:09.430517 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:09.430326 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/060aaaff-87eb-4800-99af-122329708c89-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46\" (UID: \"060aaaff-87eb-4800-99af-122329708c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:41:09.430517 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:09.430457 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lxzc\" (UniqueName: \"kubernetes.io/projected/060aaaff-87eb-4800-99af-122329708c89-kube-api-access-9lxzc\") pod \"maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46\" (UID: \"060aaaff-87eb-4800-99af-122329708c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:41:09.430631 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:09.430567 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060aaaff-87eb-4800-99af-122329708c89-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46\" (UID: \"060aaaff-87eb-4800-99af-122329708c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:41:09.468045 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:09.468001 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46"] Nov 03 22:41:09.532558 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:09.532517 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060aaaff-87eb-4800-99af-122329708c89-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46\" (UID: \"060aaaff-87eb-4800-99af-122329708c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:41:09.532754 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:09.532693 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/060aaaff-87eb-4800-99af-122329708c89-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46\" (UID: \"060aaaff-87eb-4800-99af-122329708c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:41:09.532809 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:09.532760 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/060aaaff-87eb-4800-99af-122329708c89-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46\" (UID: \"060aaaff-87eb-4800-99af-122329708c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:41:09.532860 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:09.532825 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lxzc\" (UniqueName: \"kubernetes.io/projected/060aaaff-87eb-4800-99af-122329708c89-kube-api-access-9lxzc\") pod \"maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46\" (UID: \"060aaaff-87eb-4800-99af-122329708c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:41:09.532860 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:41:09.532839 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-serving-cert: secret "maas-api-e2e-shared-b-w5-2a6801-serving-cert" not found Nov 03 22:41:09.532972 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:41:09.532924 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060aaaff-87eb-4800-99af-122329708c89-maas-api-tls podName:060aaaff-87eb-4800-99af-122329708c89 nodeName:}" failed. No retries permitted until 2026-11-03 22:41:10.032902968 +0000 UTC m=+2037.626934943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/060aaaff-87eb-4800-99af-122329708c89-maas-api-tls") pod "maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" (UID: "060aaaff-87eb-4800-99af-122329708c89") : secret "maas-api-e2e-shared-b-w5-2a6801-serving-cert" not found Nov 03 22:41:09.533593 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:09.533565 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/060aaaff-87eb-4800-99af-122329708c89-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46\" (UID: \"060aaaff-87eb-4800-99af-122329708c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:41:09.535669 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:09.535638 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060aaaff-87eb-4800-99af-122329708c89-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46\" (UID: \"060aaaff-87eb-4800-99af-122329708c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:41:09.604375 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:09.594126 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lxzc\" (UniqueName: \"kubernetes.io/projected/060aaaff-87eb-4800-99af-122329708c89-kube-api-access-9lxzc\") pod \"maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46\" (UID: \"060aaaff-87eb-4800-99af-122329708c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:41:10.056243 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:10.056202 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/060aaaff-87eb-4800-99af-122329708c89-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46\" (UID: \"060aaaff-87eb-4800-99af-122329708c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:41:10.058897 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:10.058847 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/060aaaff-87eb-4800-99af-122329708c89-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46\" (UID: \"060aaaff-87eb-4800-99af-122329708c89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:41:10.277463 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:10.277412 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:41:10.668476 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:10.668437 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46"] Nov 03 22:41:10.669251 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:41:10.669225 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060aaaff_87eb_4800_99af_122329708c89.slice/crio-f33d5fa5172b57bad439071375dd65b82e667bd9bd4f33cf634e0ba42cf115e6 WatchSource:0}: Error finding container f33d5fa5172b57bad439071375dd65b82e667bd9bd4f33cf634e0ba42cf115e6: Status 404 returned error can't find the container with id f33d5fa5172b57bad439071375dd65b82e667bd9bd4f33cf634e0ba42cf115e6 Nov 03 22:41:10.930449 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:10.930344 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" event={"ID":"060aaaff-87eb-4800-99af-122329708c89","Type":"ContainerStarted","Data":"f33d5fa5172b57bad439071375dd65b82e667bd9bd4f33cf634e0ba42cf115e6"} Nov 03 22:41:11.941314 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:11.941225 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" event={"ID":"060aaaff-87eb-4800-99af-122329708c89","Type":"ContainerStarted","Data":"015995cb6ce852319894c31d0a94c4f7afde0facbb09bbedcd19cd5fc10561e4"} Nov 03 22:41:11.941703 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:11.941342 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:41:12.028783 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:41:12.028745 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:41:12.064129 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:12.064021 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" podStartSLOduration=2.160343223 podStartE2EDuration="3.064002604s" totalImagesPullingTime="903.659381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:12.060317601 +0000 UTC m=+2039.654349577" watchObservedRunningTime="2026-11-03 22:41:12.064002604 +0000 UTC m=+2039.658034578" Nov 03 22:41:12.911656 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:12.911624 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:41:13.851465 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:13.851419 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl"] Nov 03 22:41:13.852055 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:13.851695 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" podUID="d76021af-474f-4be2-bafa-2eeb78f1144a" containerName="maas-api" containerID="cri-o://3eaaead41050c8f8aef736b9a03678cf60f9485d0daf83054cdb9c3b012a71f7" gracePeriod=30 Nov 03 22:41:14.400118 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.399184 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:41:14.440132 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.440075 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d76021af-474f-4be2-bafa-2eeb78f1144a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76021af-474f-4be2-bafa-2eeb78f1144a-maas-api-tls\") pod \"d76021af-474f-4be2-bafa-2eeb78f1144a\" (UID: \"d76021af-474f-4be2-bafa-2eeb78f1144a\") " Nov 03 22:41:14.440336 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.440202 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d76021af-474f-4be2-bafa-2eeb78f1144a-kube-api-access-lvxjj\" (UniqueName: \"kubernetes.io/projected/d76021af-474f-4be2-bafa-2eeb78f1144a-kube-api-access-lvxjj\") pod \"d76021af-474f-4be2-bafa-2eeb78f1144a\" (UID: \"d76021af-474f-4be2-bafa-2eeb78f1144a\") " Nov 03 22:41:14.440336 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.440239 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d76021af-474f-4be2-bafa-2eeb78f1144a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76021af-474f-4be2-bafa-2eeb78f1144a-openshift-service-ca\") pod \"d76021af-474f-4be2-bafa-2eeb78f1144a\" (UID: \"d76021af-474f-4be2-bafa-2eeb78f1144a\") " Nov 03 22:41:14.440336 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.440325 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d76021af-474f-4be2-bafa-2eeb78f1144a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d76021af-474f-4be2-bafa-2eeb78f1144a-metrics-certs\") pod \"d76021af-474f-4be2-bafa-2eeb78f1144a\" (UID: \"d76021af-474f-4be2-bafa-2eeb78f1144a\") " Nov 03 22:41:14.440821 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.440769 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d76021af-474f-4be2-bafa-2eeb78f1144a-openshift-service-ca" pod "d76021af-474f-4be2-bafa-2eeb78f1144a" (UID: "d76021af-474f-4be2-bafa-2eeb78f1144a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:14.441216 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.441190 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d76021af-474f-4be2-bafa-2eeb78f1144a-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:14.448303 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.448224 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d76021af-474f-4be2-bafa-2eeb78f1144a-metrics-certs" pod "d76021af-474f-4be2-bafa-2eeb78f1144a" (UID: "d76021af-474f-4be2-bafa-2eeb78f1144a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:41:14.448303 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.448245 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d76021af-474f-4be2-bafa-2eeb78f1144a-maas-api-tls" pod "d76021af-474f-4be2-bafa-2eeb78f1144a" (UID: "d76021af-474f-4be2-bafa-2eeb78f1144a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:41:14.448303 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.448254 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76021af-474f-4be2-bafa-2eeb78f1144a-kube-api-access-lvxjj" pod "d76021af-474f-4be2-bafa-2eeb78f1144a" (UID: "d76021af-474f-4be2-bafa-2eeb78f1144a"). InnerVolumeSpecName "kube-api-access-lvxjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:14.542368 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.542276 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d76021af-474f-4be2-bafa-2eeb78f1144a-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:14.542368 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.542304 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvxjj\" (UniqueName: \"kubernetes.io/projected/d76021af-474f-4be2-bafa-2eeb78f1144a-kube-api-access-lvxjj\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:14.542368 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.542313 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d76021af-474f-4be2-bafa-2eeb78f1144a-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:14.978799 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.978693 5084 generic.go:356] "Generic (PLEG): container finished" podID="d76021af-474f-4be2-bafa-2eeb78f1144a" containerID="3eaaead41050c8f8aef736b9a03678cf60f9485d0daf83054cdb9c3b012a71f7" exitCode=0 Nov 03 22:41:14.978799 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.978770 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" event={"ID":"d76021af-474f-4be2-bafa-2eeb78f1144a","Type":"ContainerDied","Data":"3eaaead41050c8f8aef736b9a03678cf60f9485d0daf83054cdb9c3b012a71f7"} Nov 03 22:41:14.979455 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.978826 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" event={"ID":"d76021af-474f-4be2-bafa-2eeb78f1144a","Type":"ContainerDied","Data":"4e86e7f5a3c34a261e562bd62e444daf1cbfe0f93f6808efe80e9c7a017bf5b5"} Nov 03 22:41:14.979455 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.978851 5084 scope.go:122] "RemoveContainer" containerID="3eaaead41050c8f8aef736b9a03678cf60f9485d0daf83054cdb9c3b012a71f7" Nov 03 22:41:14.979455 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.979010 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl" Nov 03 22:41:14.992210 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.992183 5084 scope.go:122] "RemoveContainer" containerID="3eaaead41050c8f8aef736b9a03678cf60f9485d0daf83054cdb9c3b012a71f7" Nov 03 22:41:14.992676 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:41:14.992611 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eaaead41050c8f8aef736b9a03678cf60f9485d0daf83054cdb9c3b012a71f7\": container with ID starting with 3eaaead41050c8f8aef736b9a03678cf60f9485d0daf83054cdb9c3b012a71f7 not found: ID does not exist" containerID="3eaaead41050c8f8aef736b9a03678cf60f9485d0daf83054cdb9c3b012a71f7" Nov 03 22:41:14.992822 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:14.992682 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eaaead41050c8f8aef736b9a03678cf60f9485d0daf83054cdb9c3b012a71f7"} err="failed to get container status \"3eaaead41050c8f8aef736b9a03678cf60f9485d0daf83054cdb9c3b012a71f7\": rpc error: code = NotFound desc = could not find container \"3eaaead41050c8f8aef736b9a03678cf60f9485d0daf83054cdb9c3b012a71f7\": container with ID starting with 3eaaead41050c8f8aef736b9a03678cf60f9485d0daf83054cdb9c3b012a71f7 not found: ID does not exist" Nov 03 22:41:15.149702 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:15.149658 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl"] Nov 03 22:41:15.199965 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:15.199924 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-37b70b-54d459466c-jh2vl"] Nov 03 22:41:16.472723 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:16.472683 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb"] Nov 03 22:41:16.589149 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:16.589112 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr"] Nov 03 22:41:16.632605 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:16.632524 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d76021af-474f-4be2-bafa-2eeb78f1144a" path="/var/lib/kubelet/pods/d76021af-474f-4be2-bafa-2eeb78f1144a/volumes" Nov 03 22:41:17.963160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:17.963030 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:41:21.473331 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:21.473262 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb" podUID="c9ac267f-7b6a-4932-8ef1-0b860eef79db" containerName="payload-processing" containerID="cri-o://a7ec374e8cf6338e5fbe2ff4f2b9683f99b72379d7ef7f9889b9ccc5cd068925" gracePeriod=25 Nov 03 22:41:21.589564 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:21.589517 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr" podUID="25ef43d9-40b4-49e3-a8d7-e0d6cc214394" containerName="payload-pre-processing" containerID="cri-o://28cfc8445d395200bc36e3abee7827d45cbc749bc92d36256502821ec5df836a" gracePeriod=25 Nov 03 22:41:21.818926 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:21.818892 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb" Nov 03 22:41:22.108120 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.087202 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9ac267f-7b6a-4932-8ef1-0b860eef79db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ac267f-7b6a-4932-8ef1-0b860eef79db-plugins-config-volume\") pod \"c9ac267f-7b6a-4932-8ef1-0b860eef79db\" (UID: \"c9ac267f-7b6a-4932-8ef1-0b860eef79db\") " Nov 03 22:41:22.108120 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.087543 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ac267f-7b6a-4932-8ef1-0b860eef79db-kube-api-access-hrx97\" (UniqueName: \"kubernetes.io/projected/c9ac267f-7b6a-4932-8ef1-0b860eef79db-kube-api-access-hrx97\") pod \"c9ac267f-7b6a-4932-8ef1-0b860eef79db\" (UID: \"c9ac267f-7b6a-4932-8ef1-0b860eef79db\") " Nov 03 22:41:22.162860 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.148694 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ac267f-7b6a-4932-8ef1-0b860eef79db-plugins-config-volume" pod "c9ac267f-7b6a-4932-8ef1-0b860eef79db" (UID: "c9ac267f-7b6a-4932-8ef1-0b860eef79db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:22.163080 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.162897 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ac267f-7b6a-4932-8ef1-0b860eef79db-kube-api-access-hrx97" pod "c9ac267f-7b6a-4932-8ef1-0b860eef79db" (UID: "c9ac267f-7b6a-4932-8ef1-0b860eef79db"). InnerVolumeSpecName "kube-api-access-hrx97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:22.184130 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.181772 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9ac267f-7b6a-4932-8ef1-0b860eef79db-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:22.268094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.261623 5084 generic.go:356] "Generic (PLEG): container finished" podID="c9ac267f-7b6a-4932-8ef1-0b860eef79db" containerID="a7ec374e8cf6338e5fbe2ff4f2b9683f99b72379d7ef7f9889b9ccc5cd068925" exitCode=0 Nov 03 22:41:22.268094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.261850 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb" event={"ID":"c9ac267f-7b6a-4932-8ef1-0b860eef79db","Type":"ContainerDied","Data":"a7ec374e8cf6338e5fbe2ff4f2b9683f99b72379d7ef7f9889b9ccc5cd068925"} Nov 03 22:41:22.268094 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.261931 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb" event={"ID":"c9ac267f-7b6a-4932-8ef1-0b860eef79db","Type":"ContainerDied","Data":"6bc14059f23d4092b19ce3aebd94969836fd494db724cf095bf9a32aa5cbf440"} Nov 03 22:41:22.275243 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.272085 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr" Nov 03 22:41:22.275243 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.272865 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb" Nov 03 22:41:22.275243 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.273360 5084 scope.go:122] "RemoveContainer" containerID="a7ec374e8cf6338e5fbe2ff4f2b9683f99b72379d7ef7f9889b9ccc5cd068925" Nov 03 22:41:22.277997 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.277951 5084 generic.go:356] "Generic (PLEG): container finished" podID="25ef43d9-40b4-49e3-a8d7-e0d6cc214394" containerID="28cfc8445d395200bc36e3abee7827d45cbc749bc92d36256502821ec5df836a" exitCode=0 Nov 03 22:41:22.278457 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.278426 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr" event={"ID":"25ef43d9-40b4-49e3-a8d7-e0d6cc214394","Type":"ContainerDied","Data":"28cfc8445d395200bc36e3abee7827d45cbc749bc92d36256502821ec5df836a"} Nov 03 22:41:22.278635 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.278612 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr" event={"ID":"25ef43d9-40b4-49e3-a8d7-e0d6cc214394","Type":"ContainerDied","Data":"a1f1b29a5bfdf1215aced355e348db1658a79387d01e0f3463f3a30f8e5c46e3"} Nov 03 22:41:22.282831 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.282800 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrx97\" (UniqueName: \"kubernetes.io/projected/c9ac267f-7b6a-4932-8ef1-0b860eef79db-kube-api-access-hrx97\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:22.289804 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.289777 5084 scope.go:122] "RemoveContainer" containerID="a7ec374e8cf6338e5fbe2ff4f2b9683f99b72379d7ef7f9889b9ccc5cd068925" Nov 03 22:41:22.290216 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:41:22.290191 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7ec374e8cf6338e5fbe2ff4f2b9683f99b72379d7ef7f9889b9ccc5cd068925\": container with ID starting with a7ec374e8cf6338e5fbe2ff4f2b9683f99b72379d7ef7f9889b9ccc5cd068925 not found: ID does not exist" containerID="a7ec374e8cf6338e5fbe2ff4f2b9683f99b72379d7ef7f9889b9ccc5cd068925" Nov 03 22:41:22.290314 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.290226 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ec374e8cf6338e5fbe2ff4f2b9683f99b72379d7ef7f9889b9ccc5cd068925"} err="failed to get container status \"a7ec374e8cf6338e5fbe2ff4f2b9683f99b72379d7ef7f9889b9ccc5cd068925\": rpc error: code = NotFound desc = could not find container \"a7ec374e8cf6338e5fbe2ff4f2b9683f99b72379d7ef7f9889b9ccc5cd068925\": container with ID starting with a7ec374e8cf6338e5fbe2ff4f2b9683f99b72379d7ef7f9889b9ccc5cd068925 not found: ID does not exist" Nov 03 22:41:22.290314 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.290246 5084 scope.go:122] "RemoveContainer" containerID="28cfc8445d395200bc36e3abee7827d45cbc749bc92d36256502821ec5df836a" Nov 03 22:41:22.301635 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.301603 5084 scope.go:122] "RemoveContainer" containerID="28cfc8445d395200bc36e3abee7827d45cbc749bc92d36256502821ec5df836a" Nov 03 22:41:22.302010 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:41:22.301986 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28cfc8445d395200bc36e3abee7827d45cbc749bc92d36256502821ec5df836a\": container with ID starting with 28cfc8445d395200bc36e3abee7827d45cbc749bc92d36256502821ec5df836a not found: ID does not exist" containerID="28cfc8445d395200bc36e3abee7827d45cbc749bc92d36256502821ec5df836a" Nov 03 22:41:22.302096 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.302022 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28cfc8445d395200bc36e3abee7827d45cbc749bc92d36256502821ec5df836a"} err="failed to get container status \"28cfc8445d395200bc36e3abee7827d45cbc749bc92d36256502821ec5df836a\": rpc error: code = NotFound desc = could not find container \"28cfc8445d395200bc36e3abee7827d45cbc749bc92d36256502821ec5df836a\": container with ID starting with 28cfc8445d395200bc36e3abee7827d45cbc749bc92d36256502821ec5df836a not found: ID does not exist" Nov 03 22:41:22.384325 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.384227 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25ef43d9-40b4-49e3-a8d7-e0d6cc214394-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ef43d9-40b4-49e3-a8d7-e0d6cc214394-plugins-config-volume\") pod \"25ef43d9-40b4-49e3-a8d7-e0d6cc214394\" (UID: \"25ef43d9-40b4-49e3-a8d7-e0d6cc214394\") " Nov 03 22:41:22.384325 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.384290 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ef43d9-40b4-49e3-a8d7-e0d6cc214394-kube-api-access-d5dc5\" (UniqueName: \"kubernetes.io/projected/25ef43d9-40b4-49e3-a8d7-e0d6cc214394-kube-api-access-d5dc5\") pod \"25ef43d9-40b4-49e3-a8d7-e0d6cc214394\" (UID: \"25ef43d9-40b4-49e3-a8d7-e0d6cc214394\") " Nov 03 22:41:22.384806 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.384776 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ef43d9-40b4-49e3-a8d7-e0d6cc214394-plugins-config-volume" pod "25ef43d9-40b4-49e3-a8d7-e0d6cc214394" (UID: "25ef43d9-40b4-49e3-a8d7-e0d6cc214394"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:22.384997 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.384959 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25ef43d9-40b4-49e3-a8d7-e0d6cc214394-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:22.387221 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.387188 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ef43d9-40b4-49e3-a8d7-e0d6cc214394-kube-api-access-d5dc5" pod "25ef43d9-40b4-49e3-a8d7-e0d6cc214394" (UID: "25ef43d9-40b4-49e3-a8d7-e0d6cc214394"). InnerVolumeSpecName "kube-api-access-d5dc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:22.486453 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.486403 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5dc5\" (UniqueName: \"kubernetes.io/projected/25ef43d9-40b4-49e3-a8d7-e0d6cc214394-kube-api-access-d5dc5\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:22.506844 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.506797 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb"] Nov 03 22:41:22.513477 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.513441 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ca6853-68c55849d-z5gdb"] Nov 03 22:41:22.639293 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.638115 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9ac267f-7b6a-4932-8ef1-0b860eef79db" path="/var/lib/kubelet/pods/c9ac267f-7b6a-4932-8ef1-0b860eef79db/volumes" Nov 03 22:41:22.707052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.707002 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt"] Nov 03 22:41:22.707339 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:22.707312 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" podUID="9477ee77-a2d4-4500-9bf8-5f2b8f363928" containerName="istio-proxy" containerID="cri-o://c42fd10620bd2d31a41d6eadfff13289089085f4305cc0268af6392226b128a8" gracePeriod=30 Nov 03 22:41:23.308591 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:23.308556 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr" Nov 03 22:41:23.541089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:23.538127 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr"] Nov 03 22:41:23.569139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:23.564786 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ca6853-5477d6b89f-wdnwr"] Nov 03 22:41:24.630977 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:24.630945 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25ef43d9-40b4-49e3-a8d7-e0d6cc214394" path="/var/lib/kubelet/pods/25ef43d9-40b4-49e3-a8d7-e0d6cc214394/volumes" Nov 03 22:41:25.063743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:25.063701 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh"] Nov 03 22:41:25.064336 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:25.064210 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" podUID="1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6" containerName="maas-api" containerID="cri-o://a3597d06ddfdb8ea25fc29fc40ff4be1908f8acc8abda143ea6455d66a450220" gracePeriod=30 Nov 03 22:41:25.380994 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:25.380961 5084 generic.go:356] "Generic (PLEG): container finished" podID="1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6" containerID="a3597d06ddfdb8ea25fc29fc40ff4be1908f8acc8abda143ea6455d66a450220" exitCode=0 Nov 03 22:41:25.381194 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:25.381010 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" event={"ID":"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6","Type":"ContainerDied","Data":"a3597d06ddfdb8ea25fc29fc40ff4be1908f8acc8abda143ea6455d66a450220"} Nov 03 22:41:25.429737 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:25.429678 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:41:25.471998 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:25.471960 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-kube-api-access-87rvk\" (UniqueName: \"kubernetes.io/projected/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-kube-api-access-87rvk\") pod \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\" (UID: \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\") " Nov 03 22:41:25.472291 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:25.472182 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-maas-api-tls\") pod \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\" (UID: \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\") " Nov 03 22:41:25.472291 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:25.472214 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-metrics-certs\") pod \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\" (UID: \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\") " Nov 03 22:41:25.472416 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:25.472323 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-openshift-service-ca\") pod \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\" (UID: \"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6\") " Nov 03 22:41:25.473017 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:25.472989 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-openshift-service-ca" pod "1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6" (UID: "1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:25.474692 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:25.474666 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-metrics-certs" pod "1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6" (UID: "1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:41:25.474879 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:25.474854 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-maas-api-tls" pod "1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6" (UID: "1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:41:25.474976 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:25.474959 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-kube-api-access-87rvk" pod "1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6" (UID: "1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6"). InnerVolumeSpecName "kube-api-access-87rvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:25.587088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:25.580686 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:25.587088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:25.580756 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87rvk\" (UniqueName: \"kubernetes.io/projected/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-kube-api-access-87rvk\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:25.587088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:25.580774 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:25.587088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:25.580790 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:26.390553 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:26.390522 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" Nov 03 22:41:26.391009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:26.390561 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh" event={"ID":"1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6","Type":"ContainerDied","Data":"8bfb1a97b6c4145a5b4f8a2d324ff060c0c923b845b83403fdd3a9bd3d0873fd"} Nov 03 22:41:26.391009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:26.390626 5084 scope.go:122] "RemoveContainer" containerID="a3597d06ddfdb8ea25fc29fc40ff4be1908f8acc8abda143ea6455d66a450220" Nov 03 22:41:26.559604 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:26.559559 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh"] Nov 03 22:41:26.612833 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:26.612788 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-094053-6bd5fcf9d7-5dghh"] Nov 03 22:41:26.629705 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:26.629663 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6" path="/var/lib/kubelet/pods/1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6/volumes" Nov 03 22:41:28.029844 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.029812 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:41:28.126372 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.126319 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-workload-certs\") pod \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " Nov 03 22:41:28.126566 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.126466 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-token\" (UniqueName: \"kubernetes.io/projected/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-token\") pod \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " Nov 03 22:41:28.126566 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.126501 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-data\") pod \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " Nov 03 22:41:28.126687 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.126576 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-workload-socket\") pod \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " Nov 03 22:41:28.126687 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.126610 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-podinfo\") pod \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " Nov 03 22:41:28.126687 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.126634 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-workload-certs" pod "9477ee77-a2d4-4500-9bf8-5f2b8f363928" (UID: "9477ee77-a2d4-4500-9bf8-5f2b8f363928"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:28.126852 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.126728 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istiod-ca-cert\") pod \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " Nov 03 22:41:28.126852 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.126760 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-credential-socket\") pod \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " Nov 03 22:41:28.126852 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.126806 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-envoy\") pod \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " Nov 03 22:41:28.127011 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.126973 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9477ee77-a2d4-4500-9bf8-5f2b8f363928-kube-api-access-vntb9\" (UniqueName: \"kubernetes.io/projected/9477ee77-a2d4-4500-9bf8-5f2b8f363928-kube-api-access-vntb9\") pod \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\" (UID: \"9477ee77-a2d4-4500-9bf8-5f2b8f363928\") " Nov 03 22:41:28.127155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.126853 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-workload-socket" pod "9477ee77-a2d4-4500-9bf8-5f2b8f363928" (UID: "9477ee77-a2d4-4500-9bf8-5f2b8f363928"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:28.127155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.126902 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-data" pod "9477ee77-a2d4-4500-9bf8-5f2b8f363928" (UID: "9477ee77-a2d4-4500-9bf8-5f2b8f363928"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:28.127306 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.127278 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istiod-ca-cert" pod "9477ee77-a2d4-4500-9bf8-5f2b8f363928" (UID: "9477ee77-a2d4-4500-9bf8-5f2b8f363928"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:28.127623 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.127582 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-credential-socket" pod "9477ee77-a2d4-4500-9bf8-5f2b8f363928" (UID: "9477ee77-a2d4-4500-9bf8-5f2b8f363928"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:28.127779 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.127668 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:28.127779 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.127688 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:28.127779 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.127702 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:28.127779 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.127717 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:28.129084 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.129042 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-token" pod "9477ee77-a2d4-4500-9bf8-5f2b8f363928" (UID: "9477ee77-a2d4-4500-9bf8-5f2b8f363928"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:28.129207 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.129160 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-envoy" pod "9477ee77-a2d4-4500-9bf8-5f2b8f363928" (UID: "9477ee77-a2d4-4500-9bf8-5f2b8f363928"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:41:28.129403 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.129378 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-podinfo" pod "9477ee77-a2d4-4500-9bf8-5f2b8f363928" (UID: "9477ee77-a2d4-4500-9bf8-5f2b8f363928"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:41:28.129582 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.129557 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9477ee77-a2d4-4500-9bf8-5f2b8f363928-kube-api-access-vntb9" pod "9477ee77-a2d4-4500-9bf8-5f2b8f363928" (UID: "9477ee77-a2d4-4500-9bf8-5f2b8f363928"). InnerVolumeSpecName "kube-api-access-vntb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:28.228728 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.228678 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:28.228728 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.228725 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:28.229281 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.228741 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:28.229281 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.228762 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9477ee77-a2d4-4500-9bf8-5f2b8f363928-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:28.229281 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.228777 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vntb9\" (UniqueName: \"kubernetes.io/projected/9477ee77-a2d4-4500-9bf8-5f2b8f363928-kube-api-access-vntb9\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:28.412293 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.412258 5084 generic.go:356] "Generic (PLEG): container finished" podID="9477ee77-a2d4-4500-9bf8-5f2b8f363928" containerID="c42fd10620bd2d31a41d6eadfff13289089085f4305cc0268af6392226b128a8" exitCode=0 Nov 03 22:41:28.412522 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.412318 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" Nov 03 22:41:28.412522 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.412342 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" event={"ID":"9477ee77-a2d4-4500-9bf8-5f2b8f363928","Type":"ContainerDied","Data":"c42fd10620bd2d31a41d6eadfff13289089085f4305cc0268af6392226b128a8"} Nov 03 22:41:28.412522 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.412403 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt" event={"ID":"9477ee77-a2d4-4500-9bf8-5f2b8f363928","Type":"ContainerDied","Data":"2fb54e46be2b0bef792ef5c1c5664e63ce3647ca41c9000f22db7ac97271271d"} Nov 03 22:41:28.412522 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.412426 5084 scope.go:122] "RemoveContainer" containerID="c42fd10620bd2d31a41d6eadfff13289089085f4305cc0268af6392226b128a8" Nov 03 22:41:28.425302 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.425274 5084 scope.go:122] "RemoveContainer" containerID="2eea585f1102ee82ddc0cf10f595754f4b3e0bb1a0266baed94d81193649add7" Nov 03 22:41:28.436931 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.436891 5084 scope.go:122] "RemoveContainer" containerID="c42fd10620bd2d31a41d6eadfff13289089085f4305cc0268af6392226b128a8" Nov 03 22:41:28.437385 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:41:28.437353 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42fd10620bd2d31a41d6eadfff13289089085f4305cc0268af6392226b128a8\": container with ID starting with c42fd10620bd2d31a41d6eadfff13289089085f4305cc0268af6392226b128a8 not found: ID does not exist" containerID="c42fd10620bd2d31a41d6eadfff13289089085f4305cc0268af6392226b128a8" Nov 03 22:41:28.437488 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.437404 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42fd10620bd2d31a41d6eadfff13289089085f4305cc0268af6392226b128a8"} err="failed to get container status \"c42fd10620bd2d31a41d6eadfff13289089085f4305cc0268af6392226b128a8\": rpc error: code = NotFound desc = could not find container \"c42fd10620bd2d31a41d6eadfff13289089085f4305cc0268af6392226b128a8\": container with ID starting with c42fd10620bd2d31a41d6eadfff13289089085f4305cc0268af6392226b128a8 not found: ID does not exist" Nov 03 22:41:28.437488 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.437431 5084 scope.go:122] "RemoveContainer" containerID="2eea585f1102ee82ddc0cf10f595754f4b3e0bb1a0266baed94d81193649add7" Nov 03 22:41:28.437767 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:41:28.437739 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eea585f1102ee82ddc0cf10f595754f4b3e0bb1a0266baed94d81193649add7\": container with ID starting with 2eea585f1102ee82ddc0cf10f595754f4b3e0bb1a0266baed94d81193649add7 not found: ID does not exist" containerID="2eea585f1102ee82ddc0cf10f595754f4b3e0bb1a0266baed94d81193649add7" Nov 03 22:41:28.437891 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.437773 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eea585f1102ee82ddc0cf10f595754f4b3e0bb1a0266baed94d81193649add7"} err="failed to get container status \"2eea585f1102ee82ddc0cf10f595754f4b3e0bb1a0266baed94d81193649add7\": rpc error: code = NotFound desc = could not find container \"2eea585f1102ee82ddc0cf10f595754f4b3e0bb1a0266baed94d81193649add7\": container with ID starting with 2eea585f1102ee82ddc0cf10f595754f4b3e0bb1a0266baed94d81193649add7 not found: ID does not exist" Nov 03 22:41:28.584740 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.584689 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt"] Nov 03 22:41:28.590042 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.589999 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-37b70b-openshift-default-79798d65-8tfbt"] Nov 03 22:41:28.632999 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:28.632956 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9477ee77-a2d4-4500-9bf8-5f2b8f363928" path="/var/lib/kubelet/pods/9477ee77-a2d4-4500-9bf8-5f2b8f363928/volumes" Nov 03 22:41:42.065953 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:41:42.065904 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:41:46.184324 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:46.184279 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw"] Nov 03 22:41:46.184818 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:46.184586 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" podUID="1995b4ad-49a3-4ddd-b04f-c8b02e59cf35" containerName="maas-api" containerID="cri-o://084e9e4cef95965b632466a98a4ee91929d86c059c48abc68a802b81d6104060" gracePeriod=30 Nov 03 22:41:46.621474 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:46.621441 5084 generic.go:356] "Generic (PLEG): container finished" podID="1995b4ad-49a3-4ddd-b04f-c8b02e59cf35" containerID="084e9e4cef95965b632466a98a4ee91929d86c059c48abc68a802b81d6104060" exitCode=0 Nov 03 22:41:46.630412 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:46.630373 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" event={"ID":"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35","Type":"ContainerDied","Data":"084e9e4cef95965b632466a98a4ee91929d86c059c48abc68a802b81d6104060"} Nov 03 22:41:46.824441 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:46.824403 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:41:46.853421 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:46.853374 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-maas-api-tls\") pod \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\" (UID: \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\") " Nov 03 22:41:46.853622 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:46.853445 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-metrics-certs\") pod \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\" (UID: \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\") " Nov 03 22:41:46.853622 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:46.853486 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-openshift-service-ca\") pod \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\" (UID: \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\") " Nov 03 22:41:46.853706 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:46.853686 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-kube-api-access-hhqqx\" (UniqueName: \"kubernetes.io/projected/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-kube-api-access-hhqqx\") pod \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\" (UID: \"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35\") " Nov 03 22:41:46.854464 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:46.854432 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-openshift-service-ca" pod "1995b4ad-49a3-4ddd-b04f-c8b02e59cf35" (UID: "1995b4ad-49a3-4ddd-b04f-c8b02e59cf35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:46.862321 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:46.862270 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-metrics-certs" pod "1995b4ad-49a3-4ddd-b04f-c8b02e59cf35" (UID: "1995b4ad-49a3-4ddd-b04f-c8b02e59cf35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:41:46.862528 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:46.862421 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-kube-api-access-hhqqx" pod "1995b4ad-49a3-4ddd-b04f-c8b02e59cf35" (UID: "1995b4ad-49a3-4ddd-b04f-c8b02e59cf35"). InnerVolumeSpecName "kube-api-access-hhqqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:46.873450 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:46.873377 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-maas-api-tls" pod "1995b4ad-49a3-4ddd-b04f-c8b02e59cf35" (UID: "1995b4ad-49a3-4ddd-b04f-c8b02e59cf35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:41:46.966444 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:46.955523 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:46.966444 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:46.955567 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:46.966444 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:46.955585 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:46.966444 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:46.955594 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhqqx\" (UniqueName: \"kubernetes.io/projected/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35-kube-api-access-hhqqx\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:41:47.631706 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:47.631679 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" Nov 03 22:41:47.632219 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:47.631705 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" event={"ID":"1995b4ad-49a3-4ddd-b04f-c8b02e59cf35","Type":"ContainerDied","Data":"dad2cd82480531d9e639b631ffbc45ab86646f30f0a9e012e013aacdf199b6d9"} Nov 03 22:41:47.632219 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:47.631773 5084 scope.go:122] "RemoveContainer" containerID="084e9e4cef95965b632466a98a4ee91929d86c059c48abc68a802b81d6104060" Nov 03 22:41:47.803401 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:47.803355 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw"] Nov 03 22:41:47.807203 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:47.807167 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw"] Nov 03 22:41:48.635994 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:48.635957 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1995b4ad-49a3-4ddd-b04f-c8b02e59cf35" path="/var/lib/kubelet/pods/1995b4ad-49a3-4ddd-b04f-c8b02e59cf35/volumes" Nov 03 22:41:49.647825 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:49.647782 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f61dc0-5b4b7cc45c-7qzxw" podUID="1995b4ad-49a3-4ddd-b04f-c8b02e59cf35" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.142:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:55.457411 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.457357 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq"] Nov 03 22:41:55.458239 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.458183 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" podUID="507503d1-913a-4eba-8145-4818fc379376" containerName="istio-proxy" containerID="cri-o://0bf3c9431457a8794f050d744a39c4c03e4c4ea610ec5cfce3632b7c776e355c" gracePeriod=30 Nov 03 22:41:55.787244 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.787206 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm"] Nov 03 22:41:55.789137 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789100 5084 cpu_manager.go:414] "containerMap: removing container" podUID="25ef43d9-40b4-49e3-a8d7-e0d6cc214394" containerName="payload-pre-processing" Nov 03 22:41:55.789137 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789128 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ef43d9-40b4-49e3-a8d7-e0d6cc214394" containerName="payload-pre-processing" Nov 03 22:41:55.789397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789143 5084 cpu_manager.go:414] "containerMap: removing container" podUID="c9ac267f-7b6a-4932-8ef1-0b860eef79db" containerName="payload-processing" Nov 03 22:41:55.789397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789152 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ac267f-7b6a-4932-8ef1-0b860eef79db" containerName="payload-processing" Nov 03 22:41:55.789397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789171 5084 cpu_manager.go:414] "containerMap: removing container" podUID="9477ee77-a2d4-4500-9bf8-5f2b8f363928" containerName="istio-proxy" Nov 03 22:41:55.789397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789178 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9477ee77-a2d4-4500-9bf8-5f2b8f363928" containerName="istio-proxy" Nov 03 22:41:55.789397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789209 5084 cpu_manager.go:414] "containerMap: removing container" podUID="1995b4ad-49a3-4ddd-b04f-c8b02e59cf35" containerName="maas-api" Nov 03 22:41:55.789397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789216 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1995b4ad-49a3-4ddd-b04f-c8b02e59cf35" containerName="maas-api" Nov 03 22:41:55.789397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789229 5084 cpu_manager.go:414] "containerMap: removing container" podUID="d76021af-474f-4be2-bafa-2eeb78f1144a" containerName="maas-api" Nov 03 22:41:55.789397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789236 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d76021af-474f-4be2-bafa-2eeb78f1144a" containerName="maas-api" Nov 03 22:41:55.789397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789248 5084 cpu_manager.go:414] "containerMap: removing container" podUID="1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6" containerName="maas-api" Nov 03 22:41:55.789397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789256 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6" containerName="maas-api" Nov 03 22:41:55.789397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789267 5084 cpu_manager.go:414] "containerMap: removing container" podUID="9477ee77-a2d4-4500-9bf8-5f2b8f363928" containerName="istio-proxy" Nov 03 22:41:55.789397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789274 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9477ee77-a2d4-4500-9bf8-5f2b8f363928" containerName="istio-proxy" Nov 03 22:41:55.789981 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789462 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d8a5e32-bb39-43c1-86ab-fdbb6a5f7dd6" containerName="maas-api" Nov 03 22:41:55.789981 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789478 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="9477ee77-a2d4-4500-9bf8-5f2b8f363928" containerName="istio-proxy" Nov 03 22:41:55.789981 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789498 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ef43d9-40b4-49e3-a8d7-e0d6cc214394" containerName="payload-pre-processing" Nov 03 22:41:55.789981 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789510 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ac267f-7b6a-4932-8ef1-0b860eef79db" containerName="payload-processing" Nov 03 22:41:55.789981 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789522 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="9477ee77-a2d4-4500-9bf8-5f2b8f363928" containerName="istio-proxy" Nov 03 22:41:55.789981 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789536 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="1995b4ad-49a3-4ddd-b04f-c8b02e59cf35" containerName="maas-api" Nov 03 22:41:55.789981 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.789546 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="d76021af-474f-4be2-bafa-2eeb78f1144a" containerName="maas-api" Nov 03 22:41:55.801384 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.801337 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm"] Nov 03 22:41:55.801564 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.801531 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:55.908674 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.908634 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-2a6801-openshift-default-dockercfg-nqtbg\"" Nov 03 22:41:55.951455 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.951399 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:55.951667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.951458 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd2436f-0b88-411c-bf92-91b6305c85fd-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:55.951667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.951611 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:55.951667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.951660 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:55.951922 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.951899 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:55.951993 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.951941 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:55.952056 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.952032 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wgnf\" (UniqueName: \"kubernetes.io/projected/9cd2436f-0b88-411c-bf92-91b6305c85fd-kube-api-access-8wgnf\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:55.952211 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.952088 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:55.952269 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:55.952222 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.053457 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.053358 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.053457 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.053408 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.053457 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.053446 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wgnf\" (UniqueName: \"kubernetes.io/projected/9cd2436f-0b88-411c-bf92-91b6305c85fd-kube-api-access-8wgnf\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.053750 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.053469 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.053750 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.053514 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.053750 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.053585 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.053750 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.053607 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd2436f-0b88-411c-bf92-91b6305c85fd-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.053750 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.053656 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.054009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.053838 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.054009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.053915 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.054009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.053967 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.054009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.053920 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.054009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.053985 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.054519 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.054499 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd2436f-0b88-411c-bf92-91b6305c85fd-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.056245 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.056210 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.056523 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.056501 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.112002 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.111961 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.112210 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.112043 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wgnf\" (UniqueName: \"kubernetes.io/projected/9cd2436f-0b88-411c-bf92-91b6305c85fd-kube-api-access-8wgnf\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.123003 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.122967 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:56.581860 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.581822 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm"] Nov 03 22:41:56.583628 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:41:56.583592 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd2436f_0b88_411c_bf92_91b6305c85fd.slice/crio-19e27e65d900e6a9b26db350b1ad3ae96d422b2e4ee1ebedfaa105bf78961797 WatchSource:0}: Error finding container 19e27e65d900e6a9b26db350b1ad3ae96d422b2e4ee1ebedfaa105bf78961797: Status 404 returned error can't find the container with id 19e27e65d900e6a9b26db350b1ad3ae96d422b2e4ee1ebedfaa105bf78961797 Nov 03 22:41:56.587157 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.587106 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:41:56.587276 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.587210 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:41:56.587276 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.587256 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:41:56.791897 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.791847 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" event={"ID":"9cd2436f-0b88-411c-bf92-91b6305c85fd","Type":"ContainerStarted","Data":"f96bbdd790944113a922c0f141e9f8337d9efb476d558da0c4671875f8a2afed"} Nov 03 22:41:56.792156 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.791928 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" event={"ID":"9cd2436f-0b88-411c-bf92-91b6305c85fd","Type":"ContainerStarted","Data":"19e27e65d900e6a9b26db350b1ad3ae96d422b2e4ee1ebedfaa105bf78961797"} Nov 03 22:41:56.917464 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:56.917355 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" podStartSLOduration=1.917338539 podStartE2EDuration="1.917338539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:56.913447559 +0000 UTC m=+2084.507479534" watchObservedRunningTime="2026-11-03 22:41:56.917338539 +0000 UTC m=+2084.511370513" Nov 03 22:41:57.123860 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:57.123821 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:41:57.125744 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:57.125697 5084 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 22:41:57.125932 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:57.125773 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" podUID="9cd2436f-0b88-411c-bf92-91b6305c85fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 22:41:57.515131 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:57.515084 5084 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:41:57.515315 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:57.515153 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" podUID="507503d1-913a-4eba-8145-4818fc379376" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:41:58.132995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:58.132947 5084 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 22:41:58.134567 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:58.134528 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" podUID="9cd2436f-0b88-411c-bf92-91b6305c85fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 22:41:59.124239 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:59.124195 5084 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 22:41:59.124435 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:41:59.124284 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" podUID="9cd2436f-0b88-411c-bf92-91b6305c85fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 22:42:00.123529 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.123494 5084 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 22:42:00.123961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.123550 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" podUID="9cd2436f-0b88-411c-bf92-91b6305c85fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 22:42:00.848796 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.848765 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:42:00.851216 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.851185 5084 generic.go:356] "Generic (PLEG): container finished" podID="507503d1-913a-4eba-8145-4818fc379376" containerID="0bf3c9431457a8794f050d744a39c4c03e4c4ea610ec5cfce3632b7c776e355c" exitCode=0 Nov 03 22:42:00.851361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.851237 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" Nov 03 22:42:00.851361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.851241 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" event={"ID":"507503d1-913a-4eba-8145-4818fc379376","Type":"ContainerDied","Data":"0bf3c9431457a8794f050d744a39c4c03e4c4ea610ec5cfce3632b7c776e355c"} Nov 03 22:42:00.851441 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.851367 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq" event={"ID":"507503d1-913a-4eba-8145-4818fc379376","Type":"ContainerDied","Data":"42827eb88ccf6d77078baf53431a61a396bf2bfb444caa47477a0a07d6641762"} Nov 03 22:42:00.851441 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.851392 5084 scope.go:122] "RemoveContainer" containerID="0bf3c9431457a8794f050d744a39c4c03e4c4ea610ec5cfce3632b7c776e355c" Nov 03 22:42:00.904376 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.904234 5084 scope.go:122] "RemoveContainer" containerID="3da55bf7ce31e3848f6d4c61d2ff1fd075642f74d63ad136b9266400a53b7c7d" Nov 03 22:42:00.915804 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.915775 5084 scope.go:122] "RemoveContainer" containerID="0bf3c9431457a8794f050d744a39c4c03e4c4ea610ec5cfce3632b7c776e355c" Nov 03 22:42:00.916231 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:42:00.916208 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf3c9431457a8794f050d744a39c4c03e4c4ea610ec5cfce3632b7c776e355c\": container with ID starting with 0bf3c9431457a8794f050d744a39c4c03e4c4ea610ec5cfce3632b7c776e355c not found: ID does not exist" containerID="0bf3c9431457a8794f050d744a39c4c03e4c4ea610ec5cfce3632b7c776e355c" Nov 03 22:42:00.916333 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.916239 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf3c9431457a8794f050d744a39c4c03e4c4ea610ec5cfce3632b7c776e355c"} err="failed to get container status \"0bf3c9431457a8794f050d744a39c4c03e4c4ea610ec5cfce3632b7c776e355c\": rpc error: code = NotFound desc = could not find container \"0bf3c9431457a8794f050d744a39c4c03e4c4ea610ec5cfce3632b7c776e355c\": container with ID starting with 0bf3c9431457a8794f050d744a39c4c03e4c4ea610ec5cfce3632b7c776e355c not found: ID does not exist" Nov 03 22:42:00.916333 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.916257 5084 scope.go:122] "RemoveContainer" containerID="3da55bf7ce31e3848f6d4c61d2ff1fd075642f74d63ad136b9266400a53b7c7d" Nov 03 22:42:00.916540 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:42:00.916521 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3da55bf7ce31e3848f6d4c61d2ff1fd075642f74d63ad136b9266400a53b7c7d\": container with ID starting with 3da55bf7ce31e3848f6d4c61d2ff1fd075642f74d63ad136b9266400a53b7c7d not found: ID does not exist" containerID="3da55bf7ce31e3848f6d4c61d2ff1fd075642f74d63ad136b9266400a53b7c7d" Nov 03 22:42:00.916592 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.916553 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da55bf7ce31e3848f6d4c61d2ff1fd075642f74d63ad136b9266400a53b7c7d"} err="failed to get container status \"3da55bf7ce31e3848f6d4c61d2ff1fd075642f74d63ad136b9266400a53b7c7d\": rpc error: code = NotFound desc = could not find container \"3da55bf7ce31e3848f6d4c61d2ff1fd075642f74d63ad136b9266400a53b7c7d\": container with ID starting with 3da55bf7ce31e3848f6d4c61d2ff1fd075642f74d63ad136b9266400a53b7c7d not found: ID does not exist" Nov 03 22:42:00.945383 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.945340 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507503d1-913a-4eba-8145-4818fc379376-istio-token\" (UniqueName: \"kubernetes.io/projected/507503d1-913a-4eba-8145-4818fc379376-istio-token\") pod \"507503d1-913a-4eba-8145-4818fc379376\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " Nov 03 22:42:00.945567 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.945414 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-istio-data\") pod \"507503d1-913a-4eba-8145-4818fc379376\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " Nov 03 22:42:00.945567 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.945476 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-workload-certs\") pod \"507503d1-913a-4eba-8145-4818fc379376\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " Nov 03 22:42:00.945640 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.945613 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/507503d1-913a-4eba-8145-4818fc379376-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507503d1-913a-4eba-8145-4818fc379376-istio-podinfo\") pod \"507503d1-913a-4eba-8145-4818fc379376\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " Nov 03 22:42:00.945705 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.945678 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507503d1-913a-4eba-8145-4818fc379376-kube-api-access-2j6k2\" (UniqueName: \"kubernetes.io/projected/507503d1-913a-4eba-8145-4818fc379376-kube-api-access-2j6k2\") pod \"507503d1-913a-4eba-8145-4818fc379376\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " Nov 03 22:42:00.945799 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.945766 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-istio-data" pod "507503d1-913a-4eba-8145-4818fc379376" (UID: "507503d1-913a-4eba-8145-4818fc379376"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:00.945864 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.945783 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-workload-certs" pod "507503d1-913a-4eba-8145-4818fc379376" (UID: "507503d1-913a-4eba-8145-4818fc379376"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:00.945864 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.945814 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-credential-socket\") pod \"507503d1-913a-4eba-8145-4818fc379376\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " Nov 03 22:42:00.945983 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.945964 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-istio-envoy\") pod \"507503d1-913a-4eba-8145-4818fc379376\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " Nov 03 22:42:00.946042 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.946002 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-workload-socket\") pod \"507503d1-913a-4eba-8145-4818fc379376\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " Nov 03 22:42:00.946118 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.946082 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-credential-socket" pod "507503d1-913a-4eba-8145-4818fc379376" (UID: "507503d1-913a-4eba-8145-4818fc379376"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:00.946170 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.946152 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/507503d1-913a-4eba-8145-4818fc379376-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507503d1-913a-4eba-8145-4818fc379376-istiod-ca-cert\") pod \"507503d1-913a-4eba-8145-4818fc379376\" (UID: \"507503d1-913a-4eba-8145-4818fc379376\") " Nov 03 22:42:00.946323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.946301 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-workload-socket" pod "507503d1-913a-4eba-8145-4818fc379376" (UID: "507503d1-913a-4eba-8145-4818fc379376"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:00.946641 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.946618 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507503d1-913a-4eba-8145-4818fc379376-istiod-ca-cert" pod "507503d1-913a-4eba-8145-4818fc379376" (UID: "507503d1-913a-4eba-8145-4818fc379376"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:00.946920 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.946905 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:00.946986 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.946923 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:00.946986 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.946932 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507503d1-913a-4eba-8145-4818fc379376-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:00.946986 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.946943 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:00.946986 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.946953 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:00.948164 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.948143 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507503d1-913a-4eba-8145-4818fc379376-istio-token" pod "507503d1-913a-4eba-8145-4818fc379376" (UID: "507503d1-913a-4eba-8145-4818fc379376"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:00.948389 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.948352 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507503d1-913a-4eba-8145-4818fc379376-kube-api-access-2j6k2" pod "507503d1-913a-4eba-8145-4818fc379376" (UID: "507503d1-913a-4eba-8145-4818fc379376"). InnerVolumeSpecName "kube-api-access-2j6k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:00.949267 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.949234 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-istio-envoy" pod "507503d1-913a-4eba-8145-4818fc379376" (UID: "507503d1-913a-4eba-8145-4818fc379376"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:42:00.950854 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:00.950826 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/507503d1-913a-4eba-8145-4818fc379376-istio-podinfo" pod "507503d1-913a-4eba-8145-4818fc379376" (UID: "507503d1-913a-4eba-8145-4818fc379376"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:42:01.048895 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:01.048855 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507503d1-913a-4eba-8145-4818fc379376-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:01.048895 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:01.048896 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507503d1-913a-4eba-8145-4818fc379376-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:01.049141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:01.048912 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j6k2\" (UniqueName: \"kubernetes.io/projected/507503d1-913a-4eba-8145-4818fc379376-kube-api-access-2j6k2\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:01.049141 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:01.048927 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507503d1-913a-4eba-8145-4818fc379376-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:01.123708 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:01.123613 5084 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 22:42:01.123708 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:01.123695 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" podUID="9cd2436f-0b88-411c-bf92-91b6305c85fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 22:42:01.312431 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:01.312390 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq"] Nov 03 22:42:01.316437 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:01.316399 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f61dc0-openshift-default-9bf6ff945-rcmrq"] Nov 03 22:42:02.123982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:02.123934 5084 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 22:42:02.124455 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:02.124002 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" podUID="9cd2436f-0b88-411c-bf92-91b6305c85fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 22:42:02.630749 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:02.630700 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="507503d1-913a-4eba-8145-4818fc379376" path="/var/lib/kubelet/pods/507503d1-913a-4eba-8145-4818fc379376/volumes" Nov 03 22:42:03.129318 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:03.129283 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:42:03.129829 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:03.129820 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:42:03.130921 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:03.130894 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:42:12.027042 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:42:12.027001 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:42:13.030219 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:13.030180 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:42:13.031241 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:13.031215 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:42:16.731152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:16.719392 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr"] Nov 03 22:42:16.731152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:16.719664 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" podUID="03858435-0753-4dd7-9a58-ad06f4dbc39c" containerName="maas-api" containerID="cri-o://310eb825514cd7e6fcdedc7310daf737eaa6247e913db0fb7bb11095bd10852b" gracePeriod=30 Nov 03 22:42:17.097938 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:17.097570 5084 generic.go:356] "Generic (PLEG): container finished" podID="03858435-0753-4dd7-9a58-ad06f4dbc39c" containerID="310eb825514cd7e6fcdedc7310daf737eaa6247e913db0fb7bb11095bd10852b" exitCode=0 Nov 03 22:42:17.097938 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:17.097856 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" event={"ID":"03858435-0753-4dd7-9a58-ad06f4dbc39c","Type":"ContainerDied","Data":"310eb825514cd7e6fcdedc7310daf737eaa6247e913db0fb7bb11095bd10852b"} Nov 03 22:42:17.372253 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:17.372222 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" Nov 03 22:42:17.499479 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:17.499435 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03858435-0753-4dd7-9a58-ad06f4dbc39c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/03858435-0753-4dd7-9a58-ad06f4dbc39c-metrics-certs\") pod \"03858435-0753-4dd7-9a58-ad06f4dbc39c\" (UID: \"03858435-0753-4dd7-9a58-ad06f4dbc39c\") " Nov 03 22:42:17.499693 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:17.499534 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03858435-0753-4dd7-9a58-ad06f4dbc39c-kube-api-access-hb9jd\" (UniqueName: \"kubernetes.io/projected/03858435-0753-4dd7-9a58-ad06f4dbc39c-kube-api-access-hb9jd\") pod \"03858435-0753-4dd7-9a58-ad06f4dbc39c\" (UID: \"03858435-0753-4dd7-9a58-ad06f4dbc39c\") " Nov 03 22:42:17.499693 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:17.499638 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03858435-0753-4dd7-9a58-ad06f4dbc39c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03858435-0753-4dd7-9a58-ad06f4dbc39c-openshift-service-ca\") pod \"03858435-0753-4dd7-9a58-ad06f4dbc39c\" (UID: \"03858435-0753-4dd7-9a58-ad06f4dbc39c\") " Nov 03 22:42:17.499815 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:17.499760 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03858435-0753-4dd7-9a58-ad06f4dbc39c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03858435-0753-4dd7-9a58-ad06f4dbc39c-maas-api-tls\") pod \"03858435-0753-4dd7-9a58-ad06f4dbc39c\" (UID: \"03858435-0753-4dd7-9a58-ad06f4dbc39c\") " Nov 03 22:42:17.500117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:17.500029 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03858435-0753-4dd7-9a58-ad06f4dbc39c-openshift-service-ca" pod "03858435-0753-4dd7-9a58-ad06f4dbc39c" (UID: "03858435-0753-4dd7-9a58-ad06f4dbc39c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:17.500772 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:17.500742 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03858435-0753-4dd7-9a58-ad06f4dbc39c-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.502754 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:17.502684 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03858435-0753-4dd7-9a58-ad06f4dbc39c-maas-api-tls" pod "03858435-0753-4dd7-9a58-ad06f4dbc39c" (UID: "03858435-0753-4dd7-9a58-ad06f4dbc39c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:17.502754 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:17.502708 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03858435-0753-4dd7-9a58-ad06f4dbc39c-metrics-certs" pod "03858435-0753-4dd7-9a58-ad06f4dbc39c" (UID: "03858435-0753-4dd7-9a58-ad06f4dbc39c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:17.503492 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:17.503459 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03858435-0753-4dd7-9a58-ad06f4dbc39c-kube-api-access-hb9jd" pod "03858435-0753-4dd7-9a58-ad06f4dbc39c" (UID: "03858435-0753-4dd7-9a58-ad06f4dbc39c"). InnerVolumeSpecName "kube-api-access-hb9jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:17.602692 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:17.602584 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03858435-0753-4dd7-9a58-ad06f4dbc39c-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.605320 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:17.602755 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb9jd\" (UniqueName: \"kubernetes.io/projected/03858435-0753-4dd7-9a58-ad06f4dbc39c-kube-api-access-hb9jd\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:17.605320 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:17.602780 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03858435-0753-4dd7-9a58-ad06f4dbc39c-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:18.110194 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:18.110142 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" event={"ID":"03858435-0753-4dd7-9a58-ad06f4dbc39c","Type":"ContainerDied","Data":"31997110660c0fda519e98dfcad5041a20ae9222f8fb63185d66c6174255e551"} Nov 03 22:42:18.110722 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:18.110229 5084 scope.go:122] "RemoveContainer" containerID="310eb825514cd7e6fcdedc7310daf737eaa6247e913db0fb7bb11095bd10852b" Nov 03 22:42:18.110722 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:18.110421 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr" Nov 03 22:42:18.283644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:18.283592 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr"] Nov 03 22:42:18.332709 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:18.332672 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e7d36a-744b5b5b7d-r8lcr"] Nov 03 22:42:18.630549 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:18.630507 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03858435-0753-4dd7-9a58-ad06f4dbc39c" path="/var/lib/kubelet/pods/03858435-0753-4dd7-9a58-ad06f4dbc39c/volumes" Nov 03 22:42:25.862793 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:25.862698 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7"] Nov 03 22:42:25.863954 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:25.863926 5084 cpu_manager.go:414] "containerMap: removing container" podUID="507503d1-913a-4eba-8145-4818fc379376" containerName="istio-proxy" Nov 03 22:42:25.864133 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:25.863955 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507503d1-913a-4eba-8145-4818fc379376" containerName="istio-proxy" Nov 03 22:42:25.864133 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:25.863986 5084 cpu_manager.go:414] "containerMap: removing container" podUID="03858435-0753-4dd7-9a58-ad06f4dbc39c" containerName="maas-api" Nov 03 22:42:25.864133 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:25.863993 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03858435-0753-4dd7-9a58-ad06f4dbc39c" containerName="maas-api" Nov 03 22:42:25.864331 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:25.864235 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="03858435-0753-4dd7-9a58-ad06f4dbc39c" containerName="maas-api" Nov 03 22:42:25.864331 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:25.864262 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="507503d1-913a-4eba-8145-4818fc379376" containerName="istio-proxy" Nov 03 22:42:25.864331 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:25.864280 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="507503d1-913a-4eba-8145-4818fc379376" containerName="istio-proxy" Nov 03 22:42:25.869923 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:25.869893 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:25.982591 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:25.982537 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7"] Nov 03 22:42:26.009758 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.009696 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.009983 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.009764 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.009983 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.009881 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.010135 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.010017 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.010135 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.010118 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b09b05-12bc-4377-9b4a-325eac04f4c0-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.010250 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.010164 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.010250 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.010193 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcskd\" (UniqueName: \"kubernetes.io/projected/49b09b05-12bc-4377-9b4a-325eac04f4c0-kube-api-access-pcskd\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.010364 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.010275 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.010364 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.010332 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.111776 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.111732 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b09b05-12bc-4377-9b4a-325eac04f4c0-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.111776 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.111783 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.112156 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.111886 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcskd\" (UniqueName: \"kubernetes.io/projected/49b09b05-12bc-4377-9b4a-325eac04f4c0-kube-api-access-pcskd\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.112156 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.111930 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.112156 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.111961 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.112156 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.111989 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.112156 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.112005 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.112156 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.112109 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.112495 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.112230 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.112495 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.112426 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.112495 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.112472 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.112669 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.112581 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.112764 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.112741 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b09b05-12bc-4377-9b4a-325eac04f4c0-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.112764 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.112745 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.115100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.114995 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.115237 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.115093 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.170437 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.170386 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.190216 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.170464 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcskd\" (UniqueName: \"kubernetes.io/projected/49b09b05-12bc-4377-9b4a-325eac04f4c0-kube-api-access-pcskd\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.212084 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.211474 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:26.818451 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.818418 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7"] Nov 03 22:42:26.819902 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:42:26.819875 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b09b05_12bc_4377_9b4a_325eac04f4c0.slice/crio-64a3d305755862490f761a9dc65439c774c2717a4a16a732cc1d7df3e969f26f WatchSource:0}: Error finding container 64a3d305755862490f761a9dc65439c774c2717a4a16a732cc1d7df3e969f26f: Status 404 returned error can't find the container with id 64a3d305755862490f761a9dc65439c774c2717a4a16a732cc1d7df3e969f26f Nov 03 22:42:26.822820 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.822778 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:26.822939 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.822861 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:26.822939 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:26.822915 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:27.312448 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:27.312395 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" event={"ID":"49b09b05-12bc-4377-9b4a-325eac04f4c0","Type":"ContainerStarted","Data":"e04289dc44a39530e25dc5ce11ed1cd6cfe19ba13e7e43c0e5670c738e9d59d5"} Nov 03 22:42:27.312878 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:27.312478 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" event={"ID":"49b09b05-12bc-4377-9b4a-325eac04f4c0","Type":"ContainerStarted","Data":"64a3d305755862490f761a9dc65439c774c2717a4a16a732cc1d7df3e969f26f"} Nov 03 22:42:27.445126 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:27.445051 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" podStartSLOduration=2.445033261 podStartE2EDuration="2.445033261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:27.440158697 +0000 UTC m=+2115.034190672" watchObservedRunningTime="2026-11-03 22:42:27.445033261 +0000 UTC m=+2115.039065307" Nov 03 22:42:28.217269 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:28.217229 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:28.219073 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:28.219033 5084 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 22:42:28.219233 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:28.219119 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" podUID="49b09b05-12bc-4377-9b4a-325eac04f4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 22:42:29.217256 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:29.217213 5084 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 22:42:29.217693 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:29.217291 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" podUID="49b09b05-12bc-4377-9b4a-325eac04f4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 22:42:30.217240 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:30.217195 5084 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 22:42:30.217448 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:30.217261 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" podUID="49b09b05-12bc-4377-9b4a-325eac04f4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 22:42:31.217393 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:31.217356 5084 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 22:42:31.217592 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:31.217418 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" podUID="49b09b05-12bc-4377-9b4a-325eac04f4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 22:42:31.605913 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:31.605873 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7094-ae225a"] Nov 03 22:42:31.606579 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:31.606559 5084 cpu_manager.go:414] "containerMap: removing container" podUID="507503d1-913a-4eba-8145-4818fc379376" containerName="istio-proxy" Nov 03 22:42:31.606685 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:31.606580 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507503d1-913a-4eba-8145-4818fc379376" containerName="istio-proxy" Nov 03 22:42:31.611954 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:31.611920 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7094-ae225a" Nov 03 22:42:31.704615 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:31.704551 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7094-ae225a"] Nov 03 22:42:31.717352 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:31.717310 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jj8tk\"" Nov 03 22:42:31.717567 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:31.717523 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgfgx\" (UniqueName: \"kubernetes.io/projected/308942ef-765c-40e7-a2b5-a0fb6bc20ed4-kube-api-access-bgfgx\") pod \"test-curl-7094-ae225a\" (UID: \"308942ef-765c-40e7-a2b5-a0fb6bc20ed4\") " pod="opendatahub/test-curl-7094-ae225a" Nov 03 22:42:31.819335 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:31.819290 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgfgx\" (UniqueName: \"kubernetes.io/projected/308942ef-765c-40e7-a2b5-a0fb6bc20ed4-kube-api-access-bgfgx\") pod \"test-curl-7094-ae225a\" (UID: \"308942ef-765c-40e7-a2b5-a0fb6bc20ed4\") " pod="opendatahub/test-curl-7094-ae225a" Nov 03 22:42:31.894367 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:31.894233 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgfgx\" (UniqueName: \"kubernetes.io/projected/308942ef-765c-40e7-a2b5-a0fb6bc20ed4-kube-api-access-bgfgx\") pod \"test-curl-7094-ae225a\" (UID: \"308942ef-765c-40e7-a2b5-a0fb6bc20ed4\") " pod="opendatahub/test-curl-7094-ae225a" Nov 03 22:42:31.928771 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:31.928727 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7094-ae225a" Nov 03 22:42:32.217447 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:32.217362 5084 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 22:42:32.217447 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:32.217418 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" podUID="49b09b05-12bc-4377-9b4a-325eac04f4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 22:42:32.378089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:32.377409 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7094-ae225a"] Nov 03 22:42:33.216877 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:33.216837 5084 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 22:42:33.217117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:33.216892 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" podUID="49b09b05-12bc-4377-9b4a-325eac04f4c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 22:42:33.449055 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:33.449019 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7094-ae225a" event={"ID":"308942ef-765c-40e7-a2b5-a0fb6bc20ed4","Type":"ContainerStarted","Data":"ff9507bdb7988fcd2ab29a146a7e7b6e07c7e848440a134222af69c629d8b60c"} Nov 03 22:42:33.449631 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:33.449109 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7094-ae225a" event={"ID":"308942ef-765c-40e7-a2b5-a0fb6bc20ed4","Type":"ContainerStarted","Data":"273af77c4d5514172387c85c94593703bbc1f945724458a71975471cd15791d0"} Nov 03 22:42:34.640525 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:34.640490 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:34.641087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:34.641020 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:34.657255 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:34.657196 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:42:34.792443 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:34.792373 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7094-ae225a" podStartSLOduration=3.792337375 podStartE2EDuration="3.792337375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:33.574448033 +0000 UTC m=+2121.168480007" watchObservedRunningTime="2026-11-03 22:42:34.792337375 +0000 UTC m=+2122.386369350" Nov 03 22:42:35.392519 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:35.392474 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7094-ae225a"] Nov 03 22:42:35.543797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:35.532501 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7094-ae225a"] Nov 03 22:42:35.546632 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:35.546563 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7094-ae225a" podUID="308942ef-765c-40e7-a2b5-a0fb6bc20ed4" containerName="test-curl-7094-ae225a" containerID="cri-o://ff9507bdb7988fcd2ab29a146a7e7b6e07c7e848440a134222af69c629d8b60c" gracePeriod=2 Nov 03 22:42:35.600910 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:42:35.600848 5084 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7094-ae225a\" is forbidden: User \"system:node:ip-10-0-141-84.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-84.us-west-2.compute.internal' and this object" podUID="308942ef-765c-40e7-a2b5-a0fb6bc20ed4" pod="opendatahub/test-curl-7094-ae225a" Nov 03 22:42:35.831786 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:35.827571 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b"] Nov 03 22:42:35.831786 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:35.828611 5084 cpu_manager.go:414] "containerMap: removing container" podUID="308942ef-765c-40e7-a2b5-a0fb6bc20ed4" containerName="test-curl-7094-ae225a" Nov 03 22:42:35.831786 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:35.828632 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="308942ef-765c-40e7-a2b5-a0fb6bc20ed4" containerName="test-curl-7094-ae225a" Nov 03 22:42:35.831786 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:35.828842 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="308942ef-765c-40e7-a2b5-a0fb6bc20ed4" containerName="test-curl-7094-ae225a" Nov 03 22:42:35.907916 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:35.907880 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b"] Nov 03 22:42:35.909183 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:35.908222 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" Nov 03 22:42:35.982089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:35.979630 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b30a14\"" Nov 03 22:42:35.982089 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:42:35.979896 5084 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7094-ae225a\" is forbidden: User \"system:node:ip-10-0-141-84.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-84.us-west-2.compute.internal' and this object" podUID="308942ef-765c-40e7-a2b5-a0fb6bc20ed4" pod="opendatahub/test-curl-7094-ae225a" Nov 03 22:42:35.982089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:35.980291 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b30a14-dockercfg-6xnmx\"" Nov 03 22:42:36.034090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:36.020228 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be288892-c462-4214-a907-c7a2b7b75c25-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b\" (UID: \"be288892-c462-4214-a907-c7a2b7b75c25\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" Nov 03 22:42:36.034090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:36.020434 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrmkw\" (UniqueName: \"kubernetes.io/projected/be288892-c462-4214-a907-c7a2b7b75c25-kube-api-access-lrmkw\") pod \"payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b\" (UID: \"be288892-c462-4214-a907-c7a2b7b75c25\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" Nov 03 22:42:36.121909 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:36.121808 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be288892-c462-4214-a907-c7a2b7b75c25-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b\" (UID: \"be288892-c462-4214-a907-c7a2b7b75c25\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" Nov 03 22:42:36.122157 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:36.121928 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmkw\" (UniqueName: \"kubernetes.io/projected/be288892-c462-4214-a907-c7a2b7b75c25-kube-api-access-lrmkw\") pod \"payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b\" (UID: \"be288892-c462-4214-a907-c7a2b7b75c25\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" Nov 03 22:42:36.122710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:36.122681 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be288892-c462-4214-a907-c7a2b7b75c25-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b\" (UID: \"be288892-c462-4214-a907-c7a2b7b75c25\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" Nov 03 22:42:36.181125 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:36.181081 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrmkw\" (UniqueName: \"kubernetes.io/projected/be288892-c462-4214-a907-c7a2b7b75c25-kube-api-access-lrmkw\") pod \"payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b\" (UID: \"be288892-c462-4214-a907-c7a2b7b75c25\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" Nov 03 22:42:36.234404 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:36.234364 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" Nov 03 22:42:36.639672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:36.639630 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b"] Nov 03 22:42:36.640656 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:42:36.640621 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe288892_c462_4214_a907_c7a2b7b75c25.slice/crio-bd34da136fb5cc81ece59ac785ad3e35ed16b69a6e0b6bea832af454b74c027f WatchSource:0}: Error finding container bd34da136fb5cc81ece59ac785ad3e35ed16b69a6e0b6bea832af454b74c027f: Status 404 returned error can't find the container with id bd34da136fb5cc81ece59ac785ad3e35ed16b69a6e0b6bea832af454b74c027f Nov 03 22:42:37.175647 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.175602 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk"] Nov 03 22:42:37.181361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.181326 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.290204 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.290161 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk"] Nov 03 22:42:37.339840 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.339798 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.339840 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.339841 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.340084 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.339965 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.340126 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.340087 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.340168 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.340134 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.340323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.340302 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.340359 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.340339 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtfm7\" (UniqueName: \"kubernetes.io/projected/2dc0836c-8215-4302-ac4d-67da5befb2a6-kube-api-access-xtfm7\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.340423 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.340402 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc0836c-8215-4302-ac4d-67da5befb2a6-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.340461 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.340437 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.441641 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.441542 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.441641 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.441589 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.441901 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.441716 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.441901 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.441752 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtfm7\" (UniqueName: \"kubernetes.io/projected/2dc0836c-8215-4302-ac4d-67da5befb2a6-kube-api-access-xtfm7\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.441901 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.441784 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc0836c-8215-4302-ac4d-67da5befb2a6-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.441901 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.441814 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.441901 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.441874 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.441901 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.441898 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.442255 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.441977 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.442255 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.442016 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-workload-certs\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.442255 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.442140 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-data\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.442392 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.442304 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-workload-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.442464 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.442444 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-credential-socket\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.442698 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.442674 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc0836c-8215-4302-ac4d-67da5befb2a6-istiod-ca-cert\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.444532 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.444505 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-envoy\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.444716 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.444693 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-podinfo\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.501504 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.501463 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-token\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.501708 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.501583 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtfm7\" (UniqueName: \"kubernetes.io/projected/2dc0836c-8215-4302-ac4d-67da5befb2a6-kube-api-access-xtfm7\") pod \"e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.572350 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.572183 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" event={"ID":"be288892-c462-4214-a907-c7a2b7b75c25","Type":"ContainerStarted","Data":"6b4d813871421b658f3bb1f744219de8c5d4e168681a2891188c1a3e1638264a"} Nov 03 22:42:37.572350 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.572263 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" event={"ID":"be288892-c462-4214-a907-c7a2b7b75c25","Type":"ContainerStarted","Data":"bd34da136fb5cc81ece59ac785ad3e35ed16b69a6e0b6bea832af454b74c027f"} Nov 03 22:42:37.572350 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.572306 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" Nov 03 22:42:37.691347 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.691279 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" podStartSLOduration=2.69125582 podStartE2EDuration="2.69125582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:37.687853902 +0000 UTC m=+2125.281885901" watchObservedRunningTime="2026-11-03 22:42:37.69125582 +0000 UTC m=+2125.285287797" Nov 03 22:42:37.800776 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.800729 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:37.805349 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.805319 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7094-ae225a" Nov 03 22:42:37.847594 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.847553 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/308942ef-765c-40e7-a2b5-a0fb6bc20ed4-kube-api-access-bgfgx\" (UniqueName: \"kubernetes.io/projected/308942ef-765c-40e7-a2b5-a0fb6bc20ed4-kube-api-access-bgfgx\") pod \"308942ef-765c-40e7-a2b5-a0fb6bc20ed4\" (UID: \"308942ef-765c-40e7-a2b5-a0fb6bc20ed4\") " Nov 03 22:42:37.849855 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.849819 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308942ef-765c-40e7-a2b5-a0fb6bc20ed4-kube-api-access-bgfgx" pod "308942ef-765c-40e7-a2b5-a0fb6bc20ed4" (UID: "308942ef-765c-40e7-a2b5-a0fb6bc20ed4"). InnerVolumeSpecName "kube-api-access-bgfgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:37.996499 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:37.996453 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgfgx\" (UniqueName: \"kubernetes.io/projected/308942ef-765c-40e7-a2b5-a0fb6bc20ed4-kube-api-access-bgfgx\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:38.267555 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:38.267522 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk"] Nov 03 22:42:38.268579 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:42:38.268542 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc0836c_8215_4302_ac4d_67da5befb2a6.slice/crio-496142a220eee3808cac8527b4f86c5bdcfdd1e9fc42eed1b56c7aa3f7eb515d WatchSource:0}: Error finding container 496142a220eee3808cac8527b4f86c5bdcfdd1e9fc42eed1b56c7aa3f7eb515d: Status 404 returned error can't find the container with id 496142a220eee3808cac8527b4f86c5bdcfdd1e9fc42eed1b56c7aa3f7eb515d Nov 03 22:42:38.271449 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:38.271409 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:38.271579 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:38.271501 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:38.271579 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:38.271548 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:38.594207 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:38.594109 5084 generic.go:356] "Generic (PLEG): container finished" podID="308942ef-765c-40e7-a2b5-a0fb6bc20ed4" containerID="ff9507bdb7988fcd2ab29a146a7e7b6e07c7e848440a134222af69c629d8b60c" exitCode=137 Nov 03 22:42:38.594207 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:38.594132 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7094-ae225a" Nov 03 22:42:38.594451 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:38.594151 5084 scope.go:122] "RemoveContainer" containerID="ff9507bdb7988fcd2ab29a146a7e7b6e07c7e848440a134222af69c629d8b60c" Nov 03 22:42:38.596846 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:38.596803 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" event={"ID":"2dc0836c-8215-4302-ac4d-67da5befb2a6","Type":"ContainerStarted","Data":"309de7a6c5d1b32b213b2c10a24e21a26e98d13bf5207a841888df6fcf849b22"} Nov 03 22:42:38.596982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:38.596919 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" event={"ID":"2dc0836c-8215-4302-ac4d-67da5befb2a6","Type":"ContainerStarted","Data":"496142a220eee3808cac8527b4f86c5bdcfdd1e9fc42eed1b56c7aa3f7eb515d"} Nov 03 22:42:38.606057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:38.606031 5084 scope.go:122] "RemoveContainer" containerID="ff9507bdb7988fcd2ab29a146a7e7b6e07c7e848440a134222af69c629d8b60c" Nov 03 22:42:38.606409 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:42:38.606386 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff9507bdb7988fcd2ab29a146a7e7b6e07c7e848440a134222af69c629d8b60c\": container with ID starting with ff9507bdb7988fcd2ab29a146a7e7b6e07c7e848440a134222af69c629d8b60c not found: ID does not exist" containerID="ff9507bdb7988fcd2ab29a146a7e7b6e07c7e848440a134222af69c629d8b60c" Nov 03 22:42:38.606457 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:38.606421 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff9507bdb7988fcd2ab29a146a7e7b6e07c7e848440a134222af69c629d8b60c"} err="failed to get container status \"ff9507bdb7988fcd2ab29a146a7e7b6e07c7e848440a134222af69c629d8b60c\": rpc error: code = NotFound desc = could not find container \"ff9507bdb7988fcd2ab29a146a7e7b6e07c7e848440a134222af69c629d8b60c\": container with ID starting with ff9507bdb7988fcd2ab29a146a7e7b6e07c7e848440a134222af69c629d8b60c not found: ID does not exist" Nov 03 22:42:38.632973 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:38.632936 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="308942ef-765c-40e7-a2b5-a0fb6bc20ed4" path="/var/lib/kubelet/pods/308942ef-765c-40e7-a2b5-a0fb6bc20ed4/volumes" Nov 03 22:42:38.717708 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:38.717643 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" podStartSLOduration=1.717621103 podStartE2EDuration="1.717621103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:38.716661428 +0000 UTC m=+2126.310693404" watchObservedRunningTime="2026-11-03 22:42:38.717621103 +0000 UTC m=+2126.311653078" Nov 03 22:42:38.801550 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:38.801514 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:39.803502 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:39.803449 5084 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:42:39.803937 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:39.803551 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" podUID="2dc0836c-8215-4302-ac4d-67da5befb2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:42:40.821100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:40.819213 5084 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:42:40.821100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:40.819297 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" podUID="2dc0836c-8215-4302-ac4d-67da5befb2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:42:41.095790 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.095679 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9"] Nov 03 22:42:41.125364 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.125332 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9"] Nov 03 22:42:41.125885 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.125631 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.306658 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.306608 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.306837 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.306668 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.306837 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.306784 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.306965 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.306840 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.306965 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.306944 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.307135 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.307112 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.307208 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.307192 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.307250 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.307234 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xnbh\" (UniqueName: \"kubernetes.io/projected/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-kube-api-access-4xnbh\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.307295 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.307258 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.408370 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.408268 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.408370 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.408334 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.408370 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.408370 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xnbh\" (UniqueName: \"kubernetes.io/projected/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-kube-api-access-4xnbh\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.408674 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.408395 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.408674 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.408467 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.408674 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.408499 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.408674 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.408580 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.408674 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.408657 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.408912 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.408712 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-workload-certs\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.408912 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.408714 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.409028 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.409009 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-credential-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.409205 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.409173 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-data\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.409462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.409440 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-workload-socket\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.409554 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.409470 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istiod-ca-cert\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.411318 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.411290 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-envoy\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.412504 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.412482 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-podinfo\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.485101 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.485046 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xnbh\" (UniqueName: \"kubernetes.io/projected/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-kube-api-access-4xnbh\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.485440 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.485415 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-token\") pod \"e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.744010 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.743919 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:41.801931 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.801880 5084 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:42:41.802154 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:41.801945 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" podUID="2dc0836c-8215-4302-ac4d-67da5befb2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:42:42.125176 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:42:42.125140 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:42:42.234351 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:42.234312 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9"] Nov 03 22:42:42.290488 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:42.290438 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:42.290634 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:42.290542 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:42.290634 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:42.290591 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:42:42.801239 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:42.801190 5084 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:42:42.801460 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:42.801257 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" podUID="2dc0836c-8215-4302-ac4d-67da5befb2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:42:42.952129 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:42.951396 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" event={"ID":"7e55f64c-dc5c-40c8-ae1f-6b864da55c11","Type":"ContainerStarted","Data":"01889bd0da0d1984ada1dd7f0451f1347c2d62f6d4c27aed71ae0293c289aa4b"} Nov 03 22:42:42.952129 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:42.951468 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" event={"ID":"7e55f64c-dc5c-40c8-ae1f-6b864da55c11","Type":"ContainerStarted","Data":"c1ca2b6449d77111ebda306c098830c01e0e84c262be37e22bf311de8ddbb3f9"} Nov 03 22:42:43.138431 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:43.138314 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" podStartSLOduration=3.138297096 podStartE2EDuration="3.138297096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:43.133688458 +0000 UTC m=+2130.727720432" watchObservedRunningTime="2026-11-03 22:42:43.138297096 +0000 UTC m=+2130.732329151" Nov 03 22:42:43.744981 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:43.744945 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:43.746412 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:43.746381 5084 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:42:43.746631 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:43.746611 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" podUID="7e55f64c-dc5c-40c8-ae1f-6b864da55c11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:42:43.810095 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:43.809104 5084 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:42:43.810095 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:43.809172 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" podUID="2dc0836c-8215-4302-ac4d-67da5befb2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:42:44.512900 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:44.512531 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph"] Nov 03 22:42:44.512900 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:44.512735 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" podUID="ba7426c0-eb6f-48f6-bc42-3232fefe97a4" containerName="maas-api" containerID="cri-o://dd16872db338aa60f053d2a3b172685f4f55260e8dbf71412ceb4a5b6a34ae80" gracePeriod=30 Nov 03 22:42:44.831459 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:44.828187 5084 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:42:44.831459 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:44.828248 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" podUID="7e55f64c-dc5c-40c8-ae1f-6b864da55c11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:42:44.831459 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:44.828646 5084 patch_prober.go:29] interesting pod/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:42:44.831459 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:44.828695 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" podUID="2dc0836c-8215-4302-ac4d-67da5befb2a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:42:45.079133 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:45.079092 5084 generic.go:356] "Generic (PLEG): container finished" podID="ba7426c0-eb6f-48f6-bc42-3232fefe97a4" containerID="dd16872db338aa60f053d2a3b172685f4f55260e8dbf71412ceb4a5b6a34ae80" exitCode=0 Nov 03 22:42:45.111369 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:45.102851 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" event={"ID":"ba7426c0-eb6f-48f6-bc42-3232fefe97a4","Type":"ContainerDied","Data":"dd16872db338aa60f053d2a3b172685f4f55260e8dbf71412ceb4a5b6a34ae80"} Nov 03 22:42:45.534148 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:45.534110 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:42:45.591370 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:45.591326 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-maas-api-tls\") pod \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\" (UID: \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\") " Nov 03 22:42:45.591661 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:45.591640 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-metrics-certs\") pod \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\" (UID: \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\") " Nov 03 22:42:45.591797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:45.591781 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-openshift-service-ca\") pod \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\" (UID: \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\") " Nov 03 22:42:45.592190 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:45.592166 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-kube-api-access-9bztf\" (UniqueName: \"kubernetes.io/projected/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-kube-api-access-9bztf\") pod \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\" (UID: \"ba7426c0-eb6f-48f6-bc42-3232fefe97a4\") " Nov 03 22:42:45.594543 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:45.594505 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-openshift-service-ca" pod "ba7426c0-eb6f-48f6-bc42-3232fefe97a4" (UID: "ba7426c0-eb6f-48f6-bc42-3232fefe97a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:45.626882 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:45.626840 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-kube-api-access-9bztf" pod "ba7426c0-eb6f-48f6-bc42-3232fefe97a4" (UID: "ba7426c0-eb6f-48f6-bc42-3232fefe97a4"). InnerVolumeSpecName "kube-api-access-9bztf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:45.644478 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:45.644433 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-maas-api-tls" pod "ba7426c0-eb6f-48f6-bc42-3232fefe97a4" (UID: "ba7426c0-eb6f-48f6-bc42-3232fefe97a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:45.646384 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:45.646350 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-metrics-certs" pod "ba7426c0-eb6f-48f6-bc42-3232fefe97a4" (UID: "ba7426c0-eb6f-48f6-bc42-3232fefe97a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:45.699996 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:45.699898 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bztf\" (UniqueName: \"kubernetes.io/projected/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-kube-api-access-9bztf\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:45.702218 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:45.702187 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:45.702449 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:45.702431 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:45.702567 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:45.702553 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7426c0-eb6f-48f6-bc42-3232fefe97a4-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:45.759093 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:45.757676 5084 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:42:45.759093 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:45.757755 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" podUID="7e55f64c-dc5c-40c8-ae1f-6b864da55c11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:42:46.101911 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:46.100964 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:46.102363 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:46.102333 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:46.103340 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:46.103315 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:42:46.124606 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:46.124581 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" Nov 03 22:42:46.124863 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:46.124584 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph" event={"ID":"ba7426c0-eb6f-48f6-bc42-3232fefe97a4","Type":"ContainerDied","Data":"6108df4cfd9346050e8f5934ad3c54f9ba8a619f14ea70569132b1b8a46cfdc0"} Nov 03 22:42:46.124923 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:46.124904 5084 scope.go:122] "RemoveContainer" containerID="dd16872db338aa60f053d2a3b172685f4f55260e8dbf71412ceb4a5b6a34ae80" Nov 03 22:42:46.353295 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:46.353195 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph"] Nov 03 22:42:46.355105 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:46.355048 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a92c9e-564cf6d9b6-qgsph"] Nov 03 22:42:46.634007 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:46.633913 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba7426c0-eb6f-48f6-bc42-3232fefe97a4" path="/var/lib/kubelet/pods/ba7426c0-eb6f-48f6-bc42-3232fefe97a4/volumes" Nov 03 22:42:46.746087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:46.744576 5084 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:42:46.746087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:46.744641 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" podUID="7e55f64c-dc5c-40c8-ae1f-6b864da55c11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:42:47.745053 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:47.745016 5084 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:42:47.745623 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:47.745100 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" podUID="7e55f64c-dc5c-40c8-ae1f-6b864da55c11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:42:48.601404 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:48.601372 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" Nov 03 22:42:48.745110 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:48.745023 5084 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:42:48.745536 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:48.745140 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" podUID="7e55f64c-dc5c-40c8-ae1f-6b864da55c11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:42:49.750309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:49.750276 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:49.750752 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:49.750709 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:49.751636 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:49.751611 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:42:53.200042 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:53.199995 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd"] Nov 03 22:42:56.471213 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:56.471109 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9"] Nov 03 22:42:56.471755 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:56.471453 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" podUID="dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20" containerName="maas-api" containerID="cri-o://136cfa35edd88865aacc778b00737bb5a9ece354a12ce60fd5704c5046374ef7" gracePeriod=30 Nov 03 22:42:56.980947 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:56.980564 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:42:57.079816 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.079717 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-openshift-service-ca\") pod \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\" (UID: \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\") " Nov 03 22:42:57.080006 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.079877 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-maas-api-tls\") pod \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\" (UID: \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\") " Nov 03 22:42:57.080006 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.079970 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-metrics-certs\") pod \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\" (UID: \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\") " Nov 03 22:42:57.080202 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.080047 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-kube-api-access-zlbp9\" (UniqueName: \"kubernetes.io/projected/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-kube-api-access-zlbp9\") pod \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\" (UID: \"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20\") " Nov 03 22:42:57.080258 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.080190 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-openshift-service-ca" pod "dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20" (UID: "dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:57.080797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.080771 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:57.082778 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.082737 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-metrics-certs" pod "dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20" (UID: "dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:57.082891 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.082822 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-maas-api-tls" pod "dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20" (UID: "dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:42:57.082946 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.082927 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-kube-api-access-zlbp9" pod "dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20" (UID: "dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20"). InnerVolumeSpecName "kube-api-access-zlbp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:57.182909 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.182868 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:57.182909 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.182903 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:57.182909 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.182915 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlbp9\" (UniqueName: \"kubernetes.io/projected/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20-kube-api-access-zlbp9\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:57.563222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.553417 5084 generic.go:356] "Generic (PLEG): container finished" podID="dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20" containerID="136cfa35edd88865aacc778b00737bb5a9ece354a12ce60fd5704c5046374ef7" exitCode=0 Nov 03 22:42:57.563222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.553812 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" Nov 03 22:42:57.575824 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.575771 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" event={"ID":"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20","Type":"ContainerDied","Data":"136cfa35edd88865aacc778b00737bb5a9ece354a12ce60fd5704c5046374ef7"} Nov 03 22:42:57.576168 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.576140 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9" event={"ID":"dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20","Type":"ContainerDied","Data":"f4286b9f369a8a356b7fcc5a9a3a57a6bd714b680849156d29c05f6df5054f94"} Nov 03 22:42:57.576325 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.576310 5084 scope.go:122] "RemoveContainer" containerID="136cfa35edd88865aacc778b00737bb5a9ece354a12ce60fd5704c5046374ef7" Nov 03 22:42:57.633407 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.633373 5084 scope.go:122] "RemoveContainer" containerID="136cfa35edd88865aacc778b00737bb5a9ece354a12ce60fd5704c5046374ef7" Nov 03 22:42:57.634352 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:42:57.634321 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"136cfa35edd88865aacc778b00737bb5a9ece354a12ce60fd5704c5046374ef7\": container with ID starting with 136cfa35edd88865aacc778b00737bb5a9ece354a12ce60fd5704c5046374ef7 not found: ID does not exist" containerID="136cfa35edd88865aacc778b00737bb5a9ece354a12ce60fd5704c5046374ef7" Nov 03 22:42:57.634555 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.634526 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136cfa35edd88865aacc778b00737bb5a9ece354a12ce60fd5704c5046374ef7"} err="failed to get container status \"136cfa35edd88865aacc778b00737bb5a9ece354a12ce60fd5704c5046374ef7\": rpc error: code = NotFound desc = could not find container \"136cfa35edd88865aacc778b00737bb5a9ece354a12ce60fd5704c5046374ef7\": container with ID starting with 136cfa35edd88865aacc778b00737bb5a9ece354a12ce60fd5704c5046374ef7 not found: ID does not exist" Nov 03 22:42:57.749244 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.749199 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9"] Nov 03 22:42:57.803017 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:57.802967 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ca6853-c7dccc7f5-fc7r9"] Nov 03 22:42:58.201893 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.201810 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd" podUID="d8b839e0-50e4-43d1-8b8d-7f62b553f3a8" containerName="payload-pre-processing" containerID="cri-o://1ba45996d883c63985faff339b3e5f38e9c69133dc30d3df73c0d029a3bc6125" gracePeriod=25 Nov 03 22:42:58.220489 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.220449 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2"] Nov 03 22:42:58.221184 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.221168 5084 cpu_manager.go:414] "containerMap: removing container" podUID="ba7426c0-eb6f-48f6-bc42-3232fefe97a4" containerName="maas-api" Nov 03 22:42:58.221184 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.221184 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7426c0-eb6f-48f6-bc42-3232fefe97a4" containerName="maas-api" Nov 03 22:42:58.221341 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.221217 5084 cpu_manager.go:414] "containerMap: removing container" podUID="dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20" containerName="maas-api" Nov 03 22:42:58.221341 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.221223 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20" containerName="maas-api" Nov 03 22:42:58.221341 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.221326 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20" containerName="maas-api" Nov 03 22:42:58.221341 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.221339 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7426c0-eb6f-48f6-bc42-3232fefe97a4" containerName="maas-api" Nov 03 22:42:58.225461 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.225439 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" Nov 03 22:42:58.279628 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.279595 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f881e3-serving-cert\"" Nov 03 22:42:58.280493 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.280165 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2"] Nov 03 22:42:58.417424 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.416959 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e533f5a5-fd74-4be2-828b-5efd065720ff-maas-api-tls\") pod \"maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2\" (UID: \"e533f5a5-fd74-4be2-828b-5efd065720ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" Nov 03 22:42:58.417424 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.417040 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjckn\" (UniqueName: \"kubernetes.io/projected/e533f5a5-fd74-4be2-828b-5efd065720ff-kube-api-access-mjckn\") pod \"maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2\" (UID: \"e533f5a5-fd74-4be2-828b-5efd065720ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" Nov 03 22:42:58.417424 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.417259 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e533f5a5-fd74-4be2-828b-5efd065720ff-metrics-certs\") pod \"maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2\" (UID: \"e533f5a5-fd74-4be2-828b-5efd065720ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" Nov 03 22:42:58.417424 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.417327 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e533f5a5-fd74-4be2-828b-5efd065720ff-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2\" (UID: \"e533f5a5-fd74-4be2-828b-5efd065720ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" Nov 03 22:42:58.560476 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.559184 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e533f5a5-fd74-4be2-828b-5efd065720ff-metrics-certs\") pod \"maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2\" (UID: \"e533f5a5-fd74-4be2-828b-5efd065720ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" Nov 03 22:42:58.560476 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.559289 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e533f5a5-fd74-4be2-828b-5efd065720ff-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2\" (UID: \"e533f5a5-fd74-4be2-828b-5efd065720ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" Nov 03 22:42:58.560476 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.559416 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e533f5a5-fd74-4be2-828b-5efd065720ff-maas-api-tls\") pod \"maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2\" (UID: \"e533f5a5-fd74-4be2-828b-5efd065720ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" Nov 03 22:42:58.560476 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.559482 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjckn\" (UniqueName: \"kubernetes.io/projected/e533f5a5-fd74-4be2-828b-5efd065720ff-kube-api-access-mjckn\") pod \"maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2\" (UID: \"e533f5a5-fd74-4be2-828b-5efd065720ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" Nov 03 22:42:58.581320 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.571957 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e533f5a5-fd74-4be2-828b-5efd065720ff-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2\" (UID: \"e533f5a5-fd74-4be2-828b-5efd065720ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" Nov 03 22:42:58.632538 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.632215 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjckn\" (UniqueName: \"kubernetes.io/projected/e533f5a5-fd74-4be2-828b-5efd065720ff-kube-api-access-mjckn\") pod \"maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2\" (UID: \"e533f5a5-fd74-4be2-828b-5efd065720ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" Nov 03 22:42:58.635795 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.635728 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e533f5a5-fd74-4be2-828b-5efd065720ff-maas-api-tls\") pod \"maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2\" (UID: \"e533f5a5-fd74-4be2-828b-5efd065720ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" Nov 03 22:42:58.640335 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.640265 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e533f5a5-fd74-4be2-828b-5efd065720ff-metrics-certs\") pod \"maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2\" (UID: \"e533f5a5-fd74-4be2-828b-5efd065720ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" Nov 03 22:42:58.644190 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.643415 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd" Nov 03 22:42:58.658502 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.657446 5084 generic.go:356] "Generic (PLEG): container finished" podID="d8b839e0-50e4-43d1-8b8d-7f62b553f3a8" containerID="1ba45996d883c63985faff339b3e5f38e9c69133dc30d3df73c0d029a3bc6125" exitCode=0 Nov 03 22:42:58.669523 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.660606 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8b839e0-50e4-43d1-8b8d-7f62b553f3a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b839e0-50e4-43d1-8b8d-7f62b553f3a8-plugins-config-volume\") pod \"d8b839e0-50e4-43d1-8b8d-7f62b553f3a8\" (UID: \"d8b839e0-50e4-43d1-8b8d-7f62b553f3a8\") " Nov 03 22:42:58.669523 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.660657 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b839e0-50e4-43d1-8b8d-7f62b553f3a8-kube-api-access-j678f\" (UniqueName: \"kubernetes.io/projected/d8b839e0-50e4-43d1-8b8d-7f62b553f3a8-kube-api-access-j678f\") pod \"d8b839e0-50e4-43d1-8b8d-7f62b553f3a8\" (UID: \"d8b839e0-50e4-43d1-8b8d-7f62b553f3a8\") " Nov 03 22:42:58.669523 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.662043 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b839e0-50e4-43d1-8b8d-7f62b553f3a8-plugins-config-volume" pod "d8b839e0-50e4-43d1-8b8d-7f62b553f3a8" (UID: "d8b839e0-50e4-43d1-8b8d-7f62b553f3a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:42:58.669523 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.662820 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8b839e0-50e4-43d1-8b8d-7f62b553f3a8-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:58.670383 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.670315 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b839e0-50e4-43d1-8b8d-7f62b553f3a8-kube-api-access-j678f" pod "d8b839e0-50e4-43d1-8b8d-7f62b553f3a8" (UID: "d8b839e0-50e4-43d1-8b8d-7f62b553f3a8"). InnerVolumeSpecName "kube-api-access-j678f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:42:58.673949 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.673901 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20" path="/var/lib/kubelet/pods/dc4621b2-6d1d-4ff2-a82e-d1b7d2285f20/volumes" Nov 03 22:42:58.675187 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.674596 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd" event={"ID":"d8b839e0-50e4-43d1-8b8d-7f62b553f3a8","Type":"ContainerDied","Data":"1ba45996d883c63985faff339b3e5f38e9c69133dc30d3df73c0d029a3bc6125"} Nov 03 22:42:58.675187 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.674708 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd" event={"ID":"d8b839e0-50e4-43d1-8b8d-7f62b553f3a8","Type":"ContainerDied","Data":"3a2a9e2ba54199ab45f1cd68b49c6d500ce252023c1a5f2d4372b88546e9d780"} Nov 03 22:42:58.675187 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.674749 5084 scope.go:122] "RemoveContainer" containerID="1ba45996d883c63985faff339b3e5f38e9c69133dc30d3df73c0d029a3bc6125" Nov 03 22:42:58.698284 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.696013 5084 scope.go:122] "RemoveContainer" containerID="1ba45996d883c63985faff339b3e5f38e9c69133dc30d3df73c0d029a3bc6125" Nov 03 22:42:58.698284 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:42:58.696494 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba45996d883c63985faff339b3e5f38e9c69133dc30d3df73c0d029a3bc6125\": container with ID starting with 1ba45996d883c63985faff339b3e5f38e9c69133dc30d3df73c0d029a3bc6125 not found: ID does not exist" containerID="1ba45996d883c63985faff339b3e5f38e9c69133dc30d3df73c0d029a3bc6125" Nov 03 22:42:58.698284 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.696531 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba45996d883c63985faff339b3e5f38e9c69133dc30d3df73c0d029a3bc6125"} err="failed to get container status \"1ba45996d883c63985faff339b3e5f38e9c69133dc30d3df73c0d029a3bc6125\": rpc error: code = NotFound desc = could not find container \"1ba45996d883c63985faff339b3e5f38e9c69133dc30d3df73c0d029a3bc6125\": container with ID starting with 1ba45996d883c63985faff339b3e5f38e9c69133dc30d3df73c0d029a3bc6125 not found: ID does not exist" Nov 03 22:42:58.784428 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.782953 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j678f\" (UniqueName: \"kubernetes.io/projected/d8b839e0-50e4-43d1-8b8d-7f62b553f3a8-kube-api-access-j678f\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:42:58.901409 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:58.901371 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" Nov 03 22:42:59.297113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:59.294923 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2"] Nov 03 22:42:59.297373 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:42:59.297177 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode533f5a5_fd74_4be2_828b_5efd065720ff.slice/crio-2bc4bad07363e30021d2ff8f881a4f7040149e51435a42e39e704f1f67b0b031 WatchSource:0}: Error finding container 2bc4bad07363e30021d2ff8f881a4f7040149e51435a42e39e704f1f67b0b031: Status 404 returned error can't find the container with id 2bc4bad07363e30021d2ff8f881a4f7040149e51435a42e39e704f1f67b0b031 Nov 03 22:42:59.680955 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:59.680924 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd" Nov 03 22:42:59.682978 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:59.682941 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" event={"ID":"e533f5a5-fd74-4be2-828b-5efd065720ff","Type":"ContainerStarted","Data":"2bc4bad07363e30021d2ff8f881a4f7040149e51435a42e39e704f1f67b0b031"} Nov 03 22:42:59.850549 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:59.850505 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd"] Nov 03 22:42:59.901696 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:42:59.901656 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bc88eb47-85b5495647-km7kd"] Nov 03 22:43:00.632694 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:00.632646 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8b839e0-50e4-43d1-8b8d-7f62b553f3a8" path="/var/lib/kubelet/pods/d8b839e0-50e4-43d1-8b8d-7f62b553f3a8/volumes" Nov 03 22:43:00.783090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:00.781156 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" event={"ID":"e533f5a5-fd74-4be2-828b-5efd065720ff","Type":"ContainerStarted","Data":"ae498ace186c4b137192f6d5d38495e296acfdc9e543f0458b729d25eef2abf9"} Nov 03 22:43:00.811128 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:00.794920 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" Nov 03 22:43:00.920507 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:00.920360 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" podStartSLOduration=1.9560469459999998 podStartE2EDuration="2.920333047s" totalImagesPullingTime="964.286101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:00.916103983 +0000 UTC m=+2148.510135956" watchObservedRunningTime="2026-11-03 22:43:00.920333047 +0000 UTC m=+2148.514365022" Nov 03 22:43:05.809868 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:05.809825 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh"] Nov 03 22:43:05.812632 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:05.810035 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" podUID="2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" containerName="istio-proxy" containerID="cri-o://cab8ecf5f16bd477ba766c564947427b40f10590b34619e75b8eb6276bfa0a72" gracePeriod=30 Nov 03 22:43:06.503033 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:06.502988 5084 patch_prober.go:29] interesting pod/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:43:06.503269 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:06.503091 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" podUID="2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:43:06.796212 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:06.796176 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" Nov 03 22:43:11.153648 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.153604 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:43:11.311218 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.311171 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-credential-socket\") pod \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " Nov 03 22:43:11.311218 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.311219 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-workload-socket\") pod \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " Nov 03 22:43:11.311482 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.311298 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-workload-certs\") pod \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " Nov 03 22:43:11.311482 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.311332 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-data\") pod \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " Nov 03 22:43:11.311482 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.311395 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istiod-ca-cert\") pod \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " Nov 03 22:43:11.311648 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.311495 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-credential-socket" pod "2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" (UID: "2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:11.311648 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.311527 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-envoy\") pod \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " Nov 03 22:43:11.311648 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.311557 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-workload-socket" pod "2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" (UID: "2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:11.311807 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.311650 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-kube-api-access-nd2bl\" (UniqueName: \"kubernetes.io/projected/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-kube-api-access-nd2bl\") pod \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " Nov 03 22:43:11.311807 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.311711 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-token\" (UniqueName: \"kubernetes.io/projected/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-token\") pod \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " Nov 03 22:43:11.311807 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.311648 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-data" pod "2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" (UID: "2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:11.311807 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.311665 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-workload-certs" pod "2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" (UID: "2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:11.311807 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.311737 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-podinfo\") pod \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\" (UID: \"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54\") " Nov 03 22:43:11.312126 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.311833 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istiod-ca-cert" pod "2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" (UID: "2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:11.313314 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.312536 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:11.313314 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.312566 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:11.313314 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.312582 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:11.313314 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.312597 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:11.313314 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.312612 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:11.314430 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.314304 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-envoy" pod "2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" (UID: "2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:11.314430 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.314319 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-podinfo" pod "2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" (UID: "2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:43:11.314615 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.314432 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-token" pod "2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" (UID: "2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:11.314615 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.314476 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-kube-api-access-nd2bl" pod "2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" (UID: "2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54"). InnerVolumeSpecName "kube-api-access-nd2bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:11.413672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.413557 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:11.413672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.413600 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd2bl\" (UniqueName: \"kubernetes.io/projected/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-kube-api-access-nd2bl\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:11.413672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.413611 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:11.413672 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.413620 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:11.926934 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.926900 5084 generic.go:356] "Generic (PLEG): container finished" podID="2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" containerID="cab8ecf5f16bd477ba766c564947427b40f10590b34619e75b8eb6276bfa0a72" exitCode=0 Nov 03 22:43:11.927189 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.926947 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" Nov 03 22:43:11.927189 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.926965 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" event={"ID":"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54","Type":"ContainerDied","Data":"cab8ecf5f16bd477ba766c564947427b40f10590b34619e75b8eb6276bfa0a72"} Nov 03 22:43:11.927189 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.927013 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh" event={"ID":"2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54","Type":"ContainerDied","Data":"79969c7ca07781694eecf99ef45a35e11c2f85181ca24e357b191880cb80a224"} Nov 03 22:43:11.927189 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.927040 5084 scope.go:122] "RemoveContainer" containerID="cab8ecf5f16bd477ba766c564947427b40f10590b34619e75b8eb6276bfa0a72" Nov 03 22:43:11.940280 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.940176 5084 scope.go:122] "RemoveContainer" containerID="9ebc673aa0bace8d24191831c86c3abe650f2ddd2cdcb46eb6ec59413e75b6a5" Nov 03 22:43:11.951250 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.951228 5084 scope.go:122] "RemoveContainer" containerID="cab8ecf5f16bd477ba766c564947427b40f10590b34619e75b8eb6276bfa0a72" Nov 03 22:43:11.951640 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:43:11.951614 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab8ecf5f16bd477ba766c564947427b40f10590b34619e75b8eb6276bfa0a72\": container with ID starting with cab8ecf5f16bd477ba766c564947427b40f10590b34619e75b8eb6276bfa0a72 not found: ID does not exist" containerID="cab8ecf5f16bd477ba766c564947427b40f10590b34619e75b8eb6276bfa0a72" Nov 03 22:43:11.951715 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.951650 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab8ecf5f16bd477ba766c564947427b40f10590b34619e75b8eb6276bfa0a72"} err="failed to get container status \"cab8ecf5f16bd477ba766c564947427b40f10590b34619e75b8eb6276bfa0a72\": rpc error: code = NotFound desc = could not find container \"cab8ecf5f16bd477ba766c564947427b40f10590b34619e75b8eb6276bfa0a72\": container with ID starting with cab8ecf5f16bd477ba766c564947427b40f10590b34619e75b8eb6276bfa0a72 not found: ID does not exist" Nov 03 22:43:11.951715 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.951672 5084 scope.go:122] "RemoveContainer" containerID="9ebc673aa0bace8d24191831c86c3abe650f2ddd2cdcb46eb6ec59413e75b6a5" Nov 03 22:43:11.951985 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:43:11.951961 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ebc673aa0bace8d24191831c86c3abe650f2ddd2cdcb46eb6ec59413e75b6a5\": container with ID starting with 9ebc673aa0bace8d24191831c86c3abe650f2ddd2cdcb46eb6ec59413e75b6a5 not found: ID does not exist" containerID="9ebc673aa0bace8d24191831c86c3abe650f2ddd2cdcb46eb6ec59413e75b6a5" Nov 03 22:43:11.952042 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:11.951997 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ebc673aa0bace8d24191831c86c3abe650f2ddd2cdcb46eb6ec59413e75b6a5"} err="failed to get container status \"9ebc673aa0bace8d24191831c86c3abe650f2ddd2cdcb46eb6ec59413e75b6a5\": rpc error: code = NotFound desc = could not find container \"9ebc673aa0bace8d24191831c86c3abe650f2ddd2cdcb46eb6ec59413e75b6a5\": container with ID starting with 9ebc673aa0bace8d24191831c86c3abe650f2ddd2cdcb46eb6ec59413e75b6a5 not found: ID does not exist" Nov 03 22:43:12.026360 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:43:12.026322 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:43:12.149575 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:12.149539 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh"] Nov 03 22:43:12.155673 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:12.155636 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ca6853-openshift-default-658697f489-nw2dh"] Nov 03 22:43:12.633306 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:12.633271 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" path="/var/lib/kubelet/pods/2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54/volumes" Nov 03 22:43:27.939312 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:27.939271 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz"] Nov 03 22:43:27.939776 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:27.939520 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" podUID="b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" containerName="istio-proxy" containerID="cri-o://b621cc8e9708fac69fc6f048357c0de7a270915b2207fa3bfa3eefc764b0491b" gracePeriod=30 Nov 03 22:43:29.230047 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:29.230002 5084 patch_prober.go:29] interesting pod/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:43:29.230497 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:29.230103 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" podUID="b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:43:33.143186 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.141977 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9"] Nov 03 22:43:33.143186 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.142679 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" podUID="31886473-ca58-4b81-8895-477545b480e6" containerName="maas-api" containerID="cri-o://7144e629bf0f765bb8b0989e276fc3029de9a16390e5066907cf2da555b1f60f" gracePeriod=30 Nov 03 22:43:33.239119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.238871 5084 generic.go:356] "Generic (PLEG): container finished" podID="b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" containerID="b621cc8e9708fac69fc6f048357c0de7a270915b2207fa3bfa3eefc764b0491b" exitCode=0 Nov 03 22:43:33.239310 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.239201 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" event={"ID":"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61","Type":"ContainerDied","Data":"b621cc8e9708fac69fc6f048357c0de7a270915b2207fa3bfa3eefc764b0491b"} Nov 03 22:43:33.301378 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.301347 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:43:33.362437 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.362399 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-token\" (UniqueName: \"kubernetes.io/projected/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-token\") pod \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " Nov 03 22:43:33.362749 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.362541 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-workload-socket\") pod \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " Nov 03 22:43:33.362749 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.362607 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-credential-socket\") pod \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " Nov 03 22:43:33.362749 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.362653 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-data\") pod \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " Nov 03 22:43:33.362749 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.362735 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-podinfo\") pod \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " Nov 03 22:43:33.362975 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.362818 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-kube-api-access-4f6t5\" (UniqueName: \"kubernetes.io/projected/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-kube-api-access-4f6t5\") pod \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " Nov 03 22:43:33.362975 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.362892 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istiod-ca-cert\") pod \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " Nov 03 22:43:33.362975 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.362922 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-envoy\") pod \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " Nov 03 22:43:33.363276 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.363042 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-workload-certs\") pod \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\" (UID: \"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61\") " Nov 03 22:43:33.370278 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.367421 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-workload-socket" pod "b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" (UID: "b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:33.370278 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.368213 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istiod-ca-cert" pod "b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" (UID: "b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:33.370278 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.368708 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-credential-socket" pod "b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" (UID: "b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:33.370799 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.370751 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-data" pod "b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" (UID: "b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:33.370986 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.370952 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-workload-certs" pod "b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" (UID: "b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:33.474744 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.474703 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:33.474994 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.474978 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:33.475126 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.475113 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:33.475225 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.475214 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:33.475392 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.475366 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:33.512794 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.512662 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-kube-api-access-4f6t5" pod "b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" (UID: "b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61"). InnerVolumeSpecName "kube-api-access-4f6t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:33.517142 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.516630 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-podinfo" pod "b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" (UID: "b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:43:33.517142 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.517052 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-token" pod "b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" (UID: "b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:33.517427 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.517165 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-envoy" pod "b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" (UID: "b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:33.578781 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.578638 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:33.580711 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.580679 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f6t5\" (UniqueName: \"kubernetes.io/projected/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-kube-api-access-4f6t5\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:33.580907 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.580895 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:33.580979 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.580971 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:33.600809 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.600768 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:43:33.714090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.685844 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31886473-ca58-4b81-8895-477545b480e6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31886473-ca58-4b81-8895-477545b480e6-maas-api-tls\") pod \"31886473-ca58-4b81-8895-477545b480e6\" (UID: \"31886473-ca58-4b81-8895-477545b480e6\") " Nov 03 22:43:33.714090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.686141 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31886473-ca58-4b81-8895-477545b480e6-kube-api-access-n96q7\" (UniqueName: \"kubernetes.io/projected/31886473-ca58-4b81-8895-477545b480e6-kube-api-access-n96q7\") pod \"31886473-ca58-4b81-8895-477545b480e6\" (UID: \"31886473-ca58-4b81-8895-477545b480e6\") " Nov 03 22:43:33.714090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.686196 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31886473-ca58-4b81-8895-477545b480e6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31886473-ca58-4b81-8895-477545b480e6-openshift-service-ca\") pod \"31886473-ca58-4b81-8895-477545b480e6\" (UID: \"31886473-ca58-4b81-8895-477545b480e6\") " Nov 03 22:43:33.714090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.686320 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31886473-ca58-4b81-8895-477545b480e6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/31886473-ca58-4b81-8895-477545b480e6-metrics-certs\") pod \"31886473-ca58-4b81-8895-477545b480e6\" (UID: \"31886473-ca58-4b81-8895-477545b480e6\") " Nov 03 22:43:33.714090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.688227 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31886473-ca58-4b81-8895-477545b480e6-openshift-service-ca" pod "31886473-ca58-4b81-8895-477545b480e6" (UID: "31886473-ca58-4b81-8895-477545b480e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:33.769222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.759692 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31886473-ca58-4b81-8895-477545b480e6-metrics-certs" pod "31886473-ca58-4b81-8895-477545b480e6" (UID: "31886473-ca58-4b81-8895-477545b480e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:33.769222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.759790 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31886473-ca58-4b81-8895-477545b480e6-maas-api-tls" pod "31886473-ca58-4b81-8895-477545b480e6" (UID: "31886473-ca58-4b81-8895-477545b480e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:33.769222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.760757 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31886473-ca58-4b81-8895-477545b480e6-kube-api-access-n96q7" pod "31886473-ca58-4b81-8895-477545b480e6" (UID: "31886473-ca58-4b81-8895-477545b480e6"). InnerVolumeSpecName "kube-api-access-n96q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:33.790753 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.790598 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n96q7\" (UniqueName: \"kubernetes.io/projected/31886473-ca58-4b81-8895-477545b480e6-kube-api-access-n96q7\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:33.790753 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.790636 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31886473-ca58-4b81-8895-477545b480e6-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:33.790753 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.790655 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31886473-ca58-4b81-8895-477545b480e6-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:33.790753 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:33.790669 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31886473-ca58-4b81-8895-477545b480e6-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:34.272027 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:34.271979 5084 generic.go:356] "Generic (PLEG): container finished" podID="31886473-ca58-4b81-8895-477545b480e6" containerID="7144e629bf0f765bb8b0989e276fc3029de9a16390e5066907cf2da555b1f60f" exitCode=0 Nov 03 22:43:34.273384 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:34.272904 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" Nov 03 22:43:34.277104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:34.272932 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" event={"ID":"31886473-ca58-4b81-8895-477545b480e6","Type":"ContainerDied","Data":"7144e629bf0f765bb8b0989e276fc3029de9a16390e5066907cf2da555b1f60f"} Nov 03 22:43:34.277438 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:34.277412 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9" event={"ID":"31886473-ca58-4b81-8895-477545b480e6","Type":"ContainerDied","Data":"12a390ab7fdec40294070bf25a7ad55e5b045fd263b51caec451d444e037daec"} Nov 03 22:43:34.278020 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:34.277994 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" event={"ID":"b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61","Type":"ContainerDied","Data":"ba6f036e236c586f656cc47ca757617bff45beed711c971acbca903ba7317637"} Nov 03 22:43:34.279843 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:34.276378 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz" Nov 03 22:43:34.296609 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:34.277589 5084 scope.go:122] "RemoveContainer" containerID="7144e629bf0f765bb8b0989e276fc3029de9a16390e5066907cf2da555b1f60f" Nov 03 22:43:34.382641 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:34.382084 5084 scope.go:122] "RemoveContainer" containerID="7144e629bf0f765bb8b0989e276fc3029de9a16390e5066907cf2da555b1f60f" Nov 03 22:43:34.383200 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:43:34.383042 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7144e629bf0f765bb8b0989e276fc3029de9a16390e5066907cf2da555b1f60f\": container with ID starting with 7144e629bf0f765bb8b0989e276fc3029de9a16390e5066907cf2da555b1f60f not found: ID does not exist" containerID="7144e629bf0f765bb8b0989e276fc3029de9a16390e5066907cf2da555b1f60f" Nov 03 22:43:34.383200 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:34.383150 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7144e629bf0f765bb8b0989e276fc3029de9a16390e5066907cf2da555b1f60f"} err="failed to get container status \"7144e629bf0f765bb8b0989e276fc3029de9a16390e5066907cf2da555b1f60f\": rpc error: code = NotFound desc = could not find container \"7144e629bf0f765bb8b0989e276fc3029de9a16390e5066907cf2da555b1f60f\": container with ID starting with 7144e629bf0f765bb8b0989e276fc3029de9a16390e5066907cf2da555b1f60f not found: ID does not exist" Nov 03 22:43:34.383200 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:34.383178 5084 scope.go:122] "RemoveContainer" containerID="b621cc8e9708fac69fc6f048357c0de7a270915b2207fa3bfa3eefc764b0491b" Nov 03 22:43:34.506668 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:34.506631 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9"] Nov 03 22:43:34.510771 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:34.510735 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bc88eb47-68b7d59ccf-s72k9"] Nov 03 22:43:34.631243 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:34.631157 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31886473-ca58-4b81-8895-477545b480e6" path="/var/lib/kubelet/pods/31886473-ca58-4b81-8895-477545b480e6/volumes" Nov 03 22:43:34.636129 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:34.636092 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz"] Nov 03 22:43:34.649466 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:34.649427 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f81f8f-openshift-default-99dccf577-x9rxz"] Nov 03 22:43:36.639537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:36.639495 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" path="/var/lib/kubelet/pods/b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61/volumes" Nov 03 22:43:39.730491 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.730447 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl"] Nov 03 22:43:39.731357 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.731334 5084 cpu_manager.go:414] "containerMap: removing container" podUID="2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" containerName="istio-proxy" Nov 03 22:43:39.731357 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.731359 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" containerName="istio-proxy" Nov 03 22:43:39.731501 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.731376 5084 cpu_manager.go:414] "containerMap: removing container" podUID="d8b839e0-50e4-43d1-8b8d-7f62b553f3a8" containerName="payload-pre-processing" Nov 03 22:43:39.731501 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.731384 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8b839e0-50e4-43d1-8b8d-7f62b553f3a8" containerName="payload-pre-processing" Nov 03 22:43:39.731501 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.731417 5084 cpu_manager.go:414] "containerMap: removing container" podUID="b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" containerName="istio-proxy" Nov 03 22:43:39.731501 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.731423 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" containerName="istio-proxy" Nov 03 22:43:39.731501 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.731430 5084 cpu_manager.go:414] "containerMap: removing container" podUID="31886473-ca58-4b81-8895-477545b480e6" containerName="maas-api" Nov 03 22:43:39.731501 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.731435 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31886473-ca58-4b81-8895-477545b480e6" containerName="maas-api" Nov 03 22:43:39.731501 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.731445 5084 cpu_manager.go:414] "containerMap: removing container" podUID="2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" containerName="istio-proxy" Nov 03 22:43:39.731501 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.731450 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" containerName="istio-proxy" Nov 03 22:43:39.731741 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.731549 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5ba9a52-6593-4b44-8b5a-5ed4f1fd0b61" containerName="istio-proxy" Nov 03 22:43:39.731741 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.731556 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="31886473-ca58-4b81-8895-477545b480e6" containerName="maas-api" Nov 03 22:43:39.731741 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.731563 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" containerName="istio-proxy" Nov 03 22:43:39.731741 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.731574 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8b839e0-50e4-43d1-8b8d-7f62b553f3a8" containerName="payload-pre-processing" Nov 03 22:43:39.735565 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.735538 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:43:39.783900 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.783853 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86dr4\" (UniqueName: \"kubernetes.io/projected/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-kube-api-access-86dr4\") pod \"maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl\" (UID: \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:43:39.784152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.784012 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl\" (UID: \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:43:39.784152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.784130 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-metrics-certs\") pod \"maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl\" (UID: \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:43:39.784289 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.784214 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-maas-api-tls\") pod \"maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl\" (UID: \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:43:39.789291 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.789259 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b30a14-serving-cert\"" Nov 03 22:43:39.847152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.847104 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl"] Nov 03 22:43:39.910642 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.902046 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-maas-api-tls\") pod \"maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl\" (UID: \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:43:39.910642 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.902219 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86dr4\" (UniqueName: \"kubernetes.io/projected/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-kube-api-access-86dr4\") pod \"maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl\" (UID: \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:43:39.910642 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.902338 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl\" (UID: \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:43:39.910642 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.902432 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-metrics-certs\") pod \"maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl\" (UID: \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:43:39.910642 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:43:39.903318 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-serving-cert: secret "maas-api-e2e-inf-b-b30a14-serving-cert" not found Nov 03 22:43:39.910642 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:43:39.903391 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-maas-api-tls podName:fc3ad7a3-068f-4b7b-a58a-082d9d96c7db nodeName:}" failed. No retries permitted until 2026-11-03 22:43:40.403371568 +0000 UTC m=+2187.997403534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-maas-api-tls") pod "maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" (UID: "fc3ad7a3-068f-4b7b-a58a-082d9d96c7db") : secret "maas-api-e2e-inf-b-b30a14-serving-cert" not found Nov 03 22:43:39.910642 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.903989 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl\" (UID: \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:43:39.918138 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.916912 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-metrics-certs\") pod \"maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl\" (UID: \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:43:39.962209 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:39.962173 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86dr4\" (UniqueName: \"kubernetes.io/projected/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-kube-api-access-86dr4\") pod \"maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl\" (UID: \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:43:40.420495 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:40.420439 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-maas-api-tls\") pod \"maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl\" (UID: \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:43:40.423377 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:40.423343 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-maas-api-tls\") pod \"maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl\" (UID: \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:43:40.650249 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:40.650198 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:43:41.037632 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:41.037597 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl"] Nov 03 22:43:41.038497 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:43:41.038471 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3ad7a3_068f_4b7b_a58a_082d9d96c7db.slice/crio-f654694685de7e85b98ebab9d511e3f9c6eb197540eb84b4625e93b47d73e578 WatchSource:0}: Error finding container f654694685de7e85b98ebab9d511e3f9c6eb197540eb84b4625e93b47d73e578: Status 404 returned error can't find the container with id f654694685de7e85b98ebab9d511e3f9c6eb197540eb84b4625e93b47d73e578 Nov 03 22:43:41.347988 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:41.347874 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp"] Nov 03 22:43:41.387157 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:41.387111 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" event={"ID":"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db","Type":"ContainerStarted","Data":"f654694685de7e85b98ebab9d511e3f9c6eb197540eb84b4625e93b47d73e578"} Nov 03 22:43:42.032321 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:43:42.032278 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:43:42.397352 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:42.397309 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" event={"ID":"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db","Type":"ContainerStarted","Data":"4882d2c6a3b55ea588e8a391a3ae005ff63c9caaac270a638c5324dc8a9d0a14"} Nov 03 22:43:42.397846 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:42.397558 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:43:42.519298 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:42.519217 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" podStartSLOduration=2.567352803 podStartE2EDuration="3.519198215s" totalImagesPullingTime="951.845412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:42.51592394 +0000 UTC m=+2190.109955914" watchObservedRunningTime="2026-11-03 22:43:42.519198215 +0000 UTC m=+2190.113230188" Nov 03 22:43:48.411608 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:48.411575 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:43:49.412669 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:49.412629 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll"] Nov 03 22:43:49.413484 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:49.413465 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9a6cfb-6d9f-49ee-a3e6-1a26ca23ab54" containerName="istio-proxy" Nov 03 22:43:49.417788 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:49.417764 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:43:49.471607 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:49.471558 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll"] Nov 03 22:43:49.498179 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:49.498121 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-maas-api-tls\") pod \"maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll\" (UID: \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:43:49.498406 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:49.498341 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k84h\" (UniqueName: \"kubernetes.io/projected/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-kube-api-access-9k84h\") pod \"maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll\" (UID: \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:43:49.498494 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:49.498470 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-openshift-service-ca\") pod \"maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll\" (UID: \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:43:49.498575 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:49.498558 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-metrics-certs\") pod \"maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll\" (UID: \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:43:49.518667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:49.518626 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1b2fb1-serving-cert\"" Nov 03 22:43:49.600471 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:49.600427 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k84h\" (UniqueName: \"kubernetes.io/projected/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-kube-api-access-9k84h\") pod \"maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll\" (UID: \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:43:49.600680 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:49.600516 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-openshift-service-ca\") pod \"maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll\" (UID: \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:43:49.600680 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:49.600563 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-metrics-certs\") pod \"maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll\" (UID: \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:43:49.600680 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:49.600627 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-maas-api-tls\") pod \"maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll\" (UID: \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:43:49.600826 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:43:49.600798 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-serving-cert: secret "maas-api-e2e-api-a-1b2fb1-serving-cert" not found Nov 03 22:43:49.600894 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:43:49.600883 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-maas-api-tls podName:2da4671a-8a4b-4ce4-9eb7-81183fe89ad6 nodeName:}" failed. No retries permitted until 2026-11-03 22:43:50.100867576 +0000 UTC m=+2197.694899542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-maas-api-tls") pod "maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" (UID: "2da4671a-8a4b-4ce4-9eb7-81183fe89ad6") : secret "maas-api-e2e-api-a-1b2fb1-serving-cert" not found Nov 03 22:43:49.601266 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:49.601242 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-openshift-service-ca\") pod \"maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll\" (UID: \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:43:49.603313 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:49.603288 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-metrics-certs\") pod \"maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll\" (UID: \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:43:49.707785 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:49.707694 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k84h\" (UniqueName: \"kubernetes.io/projected/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-kube-api-access-9k84h\") pod \"maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll\" (UID: \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:43:50.118964 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:50.118915 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-maas-api-tls\") pod \"maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll\" (UID: \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:43:50.121505 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:50.121471 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-maas-api-tls\") pod \"maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll\" (UID: \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:43:50.335231 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:50.335187 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:43:50.722263 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:50.721995 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll"] Nov 03 22:43:50.722674 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:43:50.722309 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da4671a_8a4b_4ce4_9eb7_81183fe89ad6.slice/crio-1f7a65fecb994e3b38411f16f1fa774f00464eb08f53bbbb5b6bc9e35ade513e WatchSource:0}: Error finding container 1f7a65fecb994e3b38411f16f1fa774f00464eb08f53bbbb5b6bc9e35ade513e: Status 404 returned error can't find the container with id 1f7a65fecb994e3b38411f16f1fa774f00464eb08f53bbbb5b6bc9e35ade513e Nov 03 22:43:51.495418 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:51.495358 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" event={"ID":"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6","Type":"ContainerStarted","Data":"1f7a65fecb994e3b38411f16f1fa774f00464eb08f53bbbb5b6bc9e35ade513e"} Nov 03 22:43:52.506119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:52.506005 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" event={"ID":"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6","Type":"ContainerStarted","Data":"ecc7bed8828154a584ce4b43f9362496923293560a32bedb2c55209391232874"} Nov 03 22:43:52.506611 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:52.506132 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:43:52.624741 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:52.624688 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" podStartSLOduration=2.702835035 podStartE2EDuration="3.62467278s" totalImagesPullingTime="921.837745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:52.623203714 +0000 UTC m=+2200.217235689" watchObservedRunningTime="2026-11-03 22:43:52.62467278 +0000 UTC m=+2200.218704772" Nov 03 22:43:56.508237 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:56.508164 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" podUID="bdf09e52-d50f-4d0e-812c-60220d5533b0" containerName="main" containerID="cri-o://460845079ddf204883639d459655b2d9c7802e11c64bcd1d97e1624aa5309782" gracePeriod=45 Nov 03 22:43:56.807375 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:56.807344 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:43:56.933988 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:56.933950 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-tmp-dir\") pod \"bdf09e52-d50f-4d0e-812c-60220d5533b0\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " Nov 03 22:43:56.934178 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:56.934014 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdf09e52-d50f-4d0e-812c-60220d5533b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf09e52-d50f-4d0e-812c-60220d5533b0-tls-certs\") pod \"bdf09e52-d50f-4d0e-812c-60220d5533b0\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " Nov 03 22:43:56.934178 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:56.934104 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-dshm\") pod \"bdf09e52-d50f-4d0e-812c-60220d5533b0\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " Nov 03 22:43:56.934178 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:56.934150 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf09e52-d50f-4d0e-812c-60220d5533b0-kube-api-access-r5w96\" (UniqueName: \"kubernetes.io/projected/bdf09e52-d50f-4d0e-812c-60220d5533b0-kube-api-access-r5w96\") pod \"bdf09e52-d50f-4d0e-812c-60220d5533b0\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " Nov 03 22:43:56.934362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:56.934183 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-home\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-home\") pod \"bdf09e52-d50f-4d0e-812c-60220d5533b0\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " Nov 03 22:43:56.934362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:56.934296 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-model-cache\") pod \"bdf09e52-d50f-4d0e-812c-60220d5533b0\" (UID: \"bdf09e52-d50f-4d0e-812c-60220d5533b0\") " Nov 03 22:43:56.934900 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:56.934872 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-model-cache" pod "bdf09e52-d50f-4d0e-812c-60220d5533b0" (UID: "bdf09e52-d50f-4d0e-812c-60220d5533b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:56.935121 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:56.935097 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-tmp-dir" pod "bdf09e52-d50f-4d0e-812c-60220d5533b0" (UID: "bdf09e52-d50f-4d0e-812c-60220d5533b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:56.941316 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:56.939247 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdf09e52-d50f-4d0e-812c-60220d5533b0-tls-certs" pod "bdf09e52-d50f-4d0e-812c-60220d5533b0" (UID: "bdf09e52-d50f-4d0e-812c-60220d5533b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:43:56.944621 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:56.944079 5084 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-tmp-dir\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:56.944621 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:56.944118 5084 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf09e52-d50f-4d0e-812c-60220d5533b0-tls-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:56.944621 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:56.944133 5084 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-model-cache\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:56.946605 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:56.946566 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-dshm" pod "bdf09e52-d50f-4d0e-812c-60220d5533b0" (UID: "bdf09e52-d50f-4d0e-812c-60220d5533b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:56.946821 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:56.946801 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-home" pod "bdf09e52-d50f-4d0e-812c-60220d5533b0" (UID: "bdf09e52-d50f-4d0e-812c-60220d5533b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:56.946894 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:56.946844 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf09e52-d50f-4d0e-812c-60220d5533b0-kube-api-access-r5w96" pod "bdf09e52-d50f-4d0e-812c-60220d5533b0" (UID: "bdf09e52-d50f-4d0e-812c-60220d5533b0"). InnerVolumeSpecName "kube-api-access-r5w96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:57.045855 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:57.045744 5084 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-dshm\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:57.045855 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:57.045803 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5w96\" (UniqueName: \"kubernetes.io/projected/bdf09e52-d50f-4d0e-812c-60220d5533b0-kube-api-access-r5w96\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:57.045855 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:57.045819 5084 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf09e52-d50f-4d0e-812c-60220d5533b0-home\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:43:57.545811 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:57.545773 5084 generic.go:356] "Generic (PLEG): container finished" podID="bdf09e52-d50f-4d0e-812c-60220d5533b0" containerID="460845079ddf204883639d459655b2d9c7802e11c64bcd1d97e1624aa5309782" exitCode=0 Nov 03 22:43:57.546271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:57.545826 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" Nov 03 22:43:57.546271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:57.545832 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" event={"ID":"bdf09e52-d50f-4d0e-812c-60220d5533b0","Type":"ContainerDied","Data":"460845079ddf204883639d459655b2d9c7802e11c64bcd1d97e1624aa5309782"} Nov 03 22:43:57.546271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:57.545935 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp" event={"ID":"bdf09e52-d50f-4d0e-812c-60220d5533b0","Type":"ContainerDied","Data":"c6c667ca1ed9b8c2a4325006c0ee9cb968bbd4c0340d3e84385eb4190b5960b8"} Nov 03 22:43:57.546271 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:57.545953 5084 scope.go:122] "RemoveContainer" containerID="460845079ddf204883639d459655b2d9c7802e11c64bcd1d97e1624aa5309782" Nov 03 22:43:57.557319 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:57.557295 5084 scope.go:122] "RemoveContainer" containerID="460845079ddf204883639d459655b2d9c7802e11c64bcd1d97e1624aa5309782" Nov 03 22:43:57.557679 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:43:57.557654 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460845079ddf204883639d459655b2d9c7802e11c64bcd1d97e1624aa5309782\": container with ID starting with 460845079ddf204883639d459655b2d9c7802e11c64bcd1d97e1624aa5309782 not found: ID does not exist" containerID="460845079ddf204883639d459655b2d9c7802e11c64bcd1d97e1624aa5309782" Nov 03 22:43:57.557758 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:57.557689 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460845079ddf204883639d459655b2d9c7802e11c64bcd1d97e1624aa5309782"} err="failed to get container status \"460845079ddf204883639d459655b2d9c7802e11c64bcd1d97e1624aa5309782\": rpc error: code = NotFound desc = could not find container \"460845079ddf204883639d459655b2d9c7802e11c64bcd1d97e1624aa5309782\": container with ID starting with 460845079ddf204883639d459655b2d9c7802e11c64bcd1d97e1624aa5309782 not found: ID does not exist" Nov 03 22:43:57.714557 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:57.714516 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp"] Nov 03 22:43:57.768749 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:57.768707 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-bc88eb47/e2e-lifecycle-model-bc88eb47-kserve-5cf4c746d5-wwmgp"] Nov 03 22:43:58.522145 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.522112 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:43:58.638892 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.638847 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdf09e52-d50f-4d0e-812c-60220d5533b0" path="/var/lib/kubelet/pods/bdf09e52-d50f-4d0e-812c-60220d5533b0/volumes" Nov 03 22:43:58.639477 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.639455 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h"] Nov 03 22:43:58.640158 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.640138 5084 cpu_manager.go:414] "containerMap: removing container" podUID="bdf09e52-d50f-4d0e-812c-60220d5533b0" containerName="main" Nov 03 22:43:58.640213 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.640160 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf09e52-d50f-4d0e-812c-60220d5533b0" containerName="main" Nov 03 22:43:58.640346 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.640333 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf09e52-d50f-4d0e-812c-60220d5533b0" containerName="main" Nov 03 22:43:58.645970 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.645941 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:43:58.695604 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.695547 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h"] Nov 03 22:43:58.701231 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.699015 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c8f168-serving-cert\"" Nov 03 22:43:58.772124 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.770922 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e76224de-abc4-45e7-8502-87f35e73f3cb-metrics-certs\") pod \"maas-api-e2e-api-a-c8f168-5c96675985-d958h\" (UID: \"e76224de-abc4-45e7-8502-87f35e73f3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:43:58.772124 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.771140 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm6cf\" (UniqueName: \"kubernetes.io/projected/e76224de-abc4-45e7-8502-87f35e73f3cb-kube-api-access-mm6cf\") pod \"maas-api-e2e-api-a-c8f168-5c96675985-d958h\" (UID: \"e76224de-abc4-45e7-8502-87f35e73f3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:43:58.772124 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.771298 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e76224de-abc4-45e7-8502-87f35e73f3cb-openshift-service-ca\") pod \"maas-api-e2e-api-a-c8f168-5c96675985-d958h\" (UID: \"e76224de-abc4-45e7-8502-87f35e73f3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:43:58.772124 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.771371 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e76224de-abc4-45e7-8502-87f35e73f3cb-maas-api-tls\") pod \"maas-api-e2e-api-a-c8f168-5c96675985-d958h\" (UID: \"e76224de-abc4-45e7-8502-87f35e73f3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:43:58.878630 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.878527 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm6cf\" (UniqueName: \"kubernetes.io/projected/e76224de-abc4-45e7-8502-87f35e73f3cb-kube-api-access-mm6cf\") pod \"maas-api-e2e-api-a-c8f168-5c96675985-d958h\" (UID: \"e76224de-abc4-45e7-8502-87f35e73f3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:43:58.914043 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.910646 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e76224de-abc4-45e7-8502-87f35e73f3cb-openshift-service-ca\") pod \"maas-api-e2e-api-a-c8f168-5c96675985-d958h\" (UID: \"e76224de-abc4-45e7-8502-87f35e73f3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:43:58.914043 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.910767 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e76224de-abc4-45e7-8502-87f35e73f3cb-maas-api-tls\") pod \"maas-api-e2e-api-a-c8f168-5c96675985-d958h\" (UID: \"e76224de-abc4-45e7-8502-87f35e73f3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:43:58.914043 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.910848 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e76224de-abc4-45e7-8502-87f35e73f3cb-metrics-certs\") pod \"maas-api-e2e-api-a-c8f168-5c96675985-d958h\" (UID: \"e76224de-abc4-45e7-8502-87f35e73f3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:43:58.914043 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.912334 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e76224de-abc4-45e7-8502-87f35e73f3cb-openshift-service-ca\") pod \"maas-api-e2e-api-a-c8f168-5c96675985-d958h\" (UID: \"e76224de-abc4-45e7-8502-87f35e73f3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:43:58.914043 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:43:58.912442 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-serving-cert: secret "maas-api-e2e-api-a-c8f168-serving-cert" not found Nov 03 22:43:58.914043 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:43:58.912511 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e76224de-abc4-45e7-8502-87f35e73f3cb-maas-api-tls podName:e76224de-abc4-45e7-8502-87f35e73f3cb nodeName:}" failed. No retries permitted until 2026-11-03 22:43:59.412488495 +0000 UTC m=+2207.006520525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e76224de-abc4-45e7-8502-87f35e73f3cb-maas-api-tls") pod "maas-api-e2e-api-a-c8f168-5c96675985-d958h" (UID: "e76224de-abc4-45e7-8502-87f35e73f3cb") : secret "maas-api-e2e-api-a-c8f168-serving-cert" not found Nov 03 22:43:58.949932 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.949902 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e76224de-abc4-45e7-8502-87f35e73f3cb-metrics-certs\") pod \"maas-api-e2e-api-a-c8f168-5c96675985-d958h\" (UID: \"e76224de-abc4-45e7-8502-87f35e73f3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:43:58.972921 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:58.972890 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm6cf\" (UniqueName: \"kubernetes.io/projected/e76224de-abc4-45e7-8502-87f35e73f3cb-kube-api-access-mm6cf\") pod \"maas-api-e2e-api-a-c8f168-5c96675985-d958h\" (UID: \"e76224de-abc4-45e7-8502-87f35e73f3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:43:59.418786 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:59.418740 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e76224de-abc4-45e7-8502-87f35e73f3cb-maas-api-tls\") pod \"maas-api-e2e-api-a-c8f168-5c96675985-d958h\" (UID: \"e76224de-abc4-45e7-8502-87f35e73f3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:43:59.421601 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:59.421564 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e76224de-abc4-45e7-8502-87f35e73f3cb-maas-api-tls\") pod \"maas-api-e2e-api-a-c8f168-5c96675985-d958h\" (UID: \"e76224de-abc4-45e7-8502-87f35e73f3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:43:59.564642 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:59.564592 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:43:59.954800 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:43:59.954761 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h"] Nov 03 22:43:59.955232 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:43:59.955206 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76224de_abc4_45e7_8502_87f35e73f3cb.slice/crio-556f6d81fa7779155897580bd27bafc94cce9bbbef960ae7fec287f6c3956aaa WatchSource:0}: Error finding container 556f6d81fa7779155897580bd27bafc94cce9bbbef960ae7fec287f6c3956aaa: Status 404 returned error can't find the container with id 556f6d81fa7779155897580bd27bafc94cce9bbbef960ae7fec287f6c3956aaa Nov 03 22:44:00.582087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:00.577836 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" event={"ID":"e76224de-abc4-45e7-8502-87f35e73f3cb","Type":"ContainerStarted","Data":"556f6d81fa7779155897580bd27bafc94cce9bbbef960ae7fec287f6c3956aaa"} Nov 03 22:44:01.598016 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:01.597973 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" event={"ID":"e76224de-abc4-45e7-8502-87f35e73f3cb","Type":"ContainerStarted","Data":"ab2d5cffd8f926ef366e82b07c42a392b574e72e13ddc0ac846e90cfdd839ad0"} Nov 03 22:44:01.600198 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:01.600171 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:44:01.717413 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:01.717351 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" podStartSLOduration=2.791560341 podStartE2EDuration="3.717333029s" totalImagesPullingTime="925.772688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:01.715217992 +0000 UTC m=+2209.309249965" watchObservedRunningTime="2026-11-03 22:44:01.717333029 +0000 UTC m=+2209.311365067" Nov 03 22:44:07.613330 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:07.613297 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:44:12.029484 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:44:12.029434 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:44:16.832521 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:16.832430 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk"] Nov 03 22:44:16.839496 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:16.839466 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:44:16.868380 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:16.868338 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b94934e9-8f2d-4b49-93f2-a386807c4c79-metrics-certs\") pod \"maas-api-e2e-api-a-97965e-658779c4d4-qp9xk\" (UID: \"b94934e9-8f2d-4b49-93f2-a386807c4c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:44:16.868587 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:16.868391 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk8tf\" (UniqueName: \"kubernetes.io/projected/b94934e9-8f2d-4b49-93f2-a386807c4c79-kube-api-access-mk8tf\") pod \"maas-api-e2e-api-a-97965e-658779c4d4-qp9xk\" (UID: \"b94934e9-8f2d-4b49-93f2-a386807c4c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:44:16.868587 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:16.868569 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94934e9-8f2d-4b49-93f2-a386807c4c79-openshift-service-ca\") pod \"maas-api-e2e-api-a-97965e-658779c4d4-qp9xk\" (UID: \"b94934e9-8f2d-4b49-93f2-a386807c4c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:44:16.868723 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:16.868687 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b94934e9-8f2d-4b49-93f2-a386807c4c79-maas-api-tls\") pod \"maas-api-e2e-api-a-97965e-658779c4d4-qp9xk\" (UID: \"b94934e9-8f2d-4b49-93f2-a386807c4c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:44:16.904479 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:16.904429 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk"] Nov 03 22:44:16.935631 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:16.935581 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-97965e-serving-cert\"" Nov 03 22:44:16.969957 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:16.969900 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b94934e9-8f2d-4b49-93f2-a386807c4c79-metrics-certs\") pod \"maas-api-e2e-api-a-97965e-658779c4d4-qp9xk\" (UID: \"b94934e9-8f2d-4b49-93f2-a386807c4c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:44:16.970273 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:16.970243 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk8tf\" (UniqueName: \"kubernetes.io/projected/b94934e9-8f2d-4b49-93f2-a386807c4c79-kube-api-access-mk8tf\") pod \"maas-api-e2e-api-a-97965e-658779c4d4-qp9xk\" (UID: \"b94934e9-8f2d-4b49-93f2-a386807c4c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:44:16.970476 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:16.970452 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94934e9-8f2d-4b49-93f2-a386807c4c79-openshift-service-ca\") pod \"maas-api-e2e-api-a-97965e-658779c4d4-qp9xk\" (UID: \"b94934e9-8f2d-4b49-93f2-a386807c4c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:44:16.970654 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:16.970639 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b94934e9-8f2d-4b49-93f2-a386807c4c79-maas-api-tls\") pod \"maas-api-e2e-api-a-97965e-658779c4d4-qp9xk\" (UID: \"b94934e9-8f2d-4b49-93f2-a386807c4c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:44:16.971086 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:44:16.971047 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-serving-cert: secret "maas-api-e2e-api-a-97965e-serving-cert" not found Nov 03 22:44:16.971271 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:44:16.971259 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b94934e9-8f2d-4b49-93f2-a386807c4c79-maas-api-tls podName:b94934e9-8f2d-4b49-93f2-a386807c4c79 nodeName:}" failed. No retries permitted until 2026-11-03 22:44:17.47123645 +0000 UTC m=+2225.065268420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b94934e9-8f2d-4b49-93f2-a386807c4c79-maas-api-tls") pod "maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" (UID: "b94934e9-8f2d-4b49-93f2-a386807c4c79") : secret "maas-api-e2e-api-a-97965e-serving-cert" not found Nov 03 22:44:16.973086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:16.973036 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94934e9-8f2d-4b49-93f2-a386807c4c79-openshift-service-ca\") pod \"maas-api-e2e-api-a-97965e-658779c4d4-qp9xk\" (UID: \"b94934e9-8f2d-4b49-93f2-a386807c4c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:44:16.988927 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:16.988828 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b94934e9-8f2d-4b49-93f2-a386807c4c79-metrics-certs\") pod \"maas-api-e2e-api-a-97965e-658779c4d4-qp9xk\" (UID: \"b94934e9-8f2d-4b49-93f2-a386807c4c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:44:17.031268 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:17.031225 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk8tf\" (UniqueName: \"kubernetes.io/projected/b94934e9-8f2d-4b49-93f2-a386807c4c79-kube-api-access-mk8tf\") pod \"maas-api-e2e-api-a-97965e-658779c4d4-qp9xk\" (UID: \"b94934e9-8f2d-4b49-93f2-a386807c4c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:44:17.499689 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:17.499646 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b94934e9-8f2d-4b49-93f2-a386807c4c79-maas-api-tls\") pod \"maas-api-e2e-api-a-97965e-658779c4d4-qp9xk\" (UID: \"b94934e9-8f2d-4b49-93f2-a386807c4c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:44:17.502401 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:17.502367 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b94934e9-8f2d-4b49-93f2-a386807c4c79-maas-api-tls\") pod \"maas-api-e2e-api-a-97965e-658779c4d4-qp9xk\" (UID: \"b94934e9-8f2d-4b49-93f2-a386807c4c79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:44:17.767156 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:17.764318 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:44:18.281129 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:18.281090 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk"] Nov 03 22:44:18.283046 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:44:18.283004 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94934e9_8f2d_4b49_93f2_a386807c4c79.slice/crio-c7122e0d44b6588d9a7a5ea4686a4a4a1ee9517bd155b9c6f272a259f37a9191 WatchSource:0}: Error finding container c7122e0d44b6588d9a7a5ea4686a4a4a1ee9517bd155b9c6f272a259f37a9191: Status 404 returned error can't find the container with id c7122e0d44b6588d9a7a5ea4686a4a4a1ee9517bd155b9c6f272a259f37a9191 Nov 03 22:44:18.964889 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:18.964840 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" event={"ID":"b94934e9-8f2d-4b49-93f2-a386807c4c79","Type":"ContainerStarted","Data":"c7122e0d44b6588d9a7a5ea4686a4a4a1ee9517bd155b9c6f272a259f37a9191"} Nov 03 22:44:19.976706 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:19.976670 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" event={"ID":"b94934e9-8f2d-4b49-93f2-a386807c4c79","Type":"ContainerStarted","Data":"30b42ec05dd7803d750eec79cfccb119d32173836502f9537412cef3e1e11f32"} Nov 03 22:44:19.977221 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:19.976738 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:44:20.098787 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:20.098721 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" podStartSLOduration=3.1335669680000002 podStartE2EDuration="4.098698832s" totalImagesPullingTime="965.131864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:20.095177192 +0000 UTC m=+2227.689209168" watchObservedRunningTime="2026-11-03 22:44:20.098698832 +0000 UTC m=+2227.692730807" Nov 03 22:44:25.067714 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:25.067671 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk"] Nov 03 22:44:25.068221 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:25.067891 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" podUID="2dc0836c-8215-4302-ac4d-67da5befb2a6" containerName="istio-proxy" containerID="cri-o://309de7a6c5d1b32b213b2c10a24e21a26e98d13bf5207a841888df6fcf849b22" gracePeriod=30 Nov 03 22:44:25.082009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:25.081974 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z"] Nov 03 22:44:25.082269 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:25.082234 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" podUID="ad18cacf-d8cc-4b0d-909e-7c60805abe2f" containerName="istio-proxy" containerID="cri-o://c109be0fb54c3468b9277ae28aedd0104fb9e2458f30cce82c2f32d1932edd3c" gracePeriod=30 Nov 03 22:44:25.644770 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:25.644727 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n"] Nov 03 22:44:25.648406 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:25.645261 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" podUID="0da90ec4-3e54-4bfc-984b-f3d60baa79ea" containerName="maas-api" containerID="cri-o://1566de9592be6f7a313bc22b5db3c7430b1d3a16f731b933bfdab41dd15d3177" gracePeriod=30 Nov 03 22:44:25.992727 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:25.992700 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:44:26.053296 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:26.053264 5084 generic.go:356] "Generic (PLEG): container finished" podID="0da90ec4-3e54-4bfc-984b-f3d60baa79ea" containerID="1566de9592be6f7a313bc22b5db3c7430b1d3a16f731b933bfdab41dd15d3177" exitCode=0 Nov 03 22:44:26.053467 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:26.053362 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" event={"ID":"0da90ec4-3e54-4bfc-984b-f3d60baa79ea","Type":"ContainerDied","Data":"1566de9592be6f7a313bc22b5db3c7430b1d3a16f731b933bfdab41dd15d3177"} Nov 03 22:44:26.433182 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:26.433147 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:44:26.598371 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:26.598326 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-openshift-service-ca\") pod \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\" (UID: \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\") " Nov 03 22:44:26.598606 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:26.598483 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-maas-api-tls\") pod \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\" (UID: \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\") " Nov 03 22:44:26.598606 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:26.598544 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-metrics-certs\") pod \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\" (UID: \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\") " Nov 03 22:44:26.598606 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:26.598594 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-kube-api-access-wqvw9\" (UniqueName: \"kubernetes.io/projected/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-kube-api-access-wqvw9\") pod \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\" (UID: \"0da90ec4-3e54-4bfc-984b-f3d60baa79ea\") " Nov 03 22:44:26.599873 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:26.599835 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-openshift-service-ca" pod "0da90ec4-3e54-4bfc-984b-f3d60baa79ea" (UID: "0da90ec4-3e54-4bfc-984b-f3d60baa79ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:26.601525 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:26.601482 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-metrics-certs" pod "0da90ec4-3e54-4bfc-984b-f3d60baa79ea" (UID: "0da90ec4-3e54-4bfc-984b-f3d60baa79ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:26.603789 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:26.603745 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-maas-api-tls" pod "0da90ec4-3e54-4bfc-984b-f3d60baa79ea" (UID: "0da90ec4-3e54-4bfc-984b-f3d60baa79ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:26.603950 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:26.603891 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-kube-api-access-wqvw9" pod "0da90ec4-3e54-4bfc-984b-f3d60baa79ea" (UID: "0da90ec4-3e54-4bfc-984b-f3d60baa79ea"). InnerVolumeSpecName "kube-api-access-wqvw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:26.712424 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:26.712387 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:26.713914 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:26.713887 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:26.718214 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:26.718168 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqvw9\" (UniqueName: \"kubernetes.io/projected/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-kube-api-access-wqvw9\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:26.718452 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:26.718434 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da90ec4-3e54-4bfc-984b-f3d60baa79ea-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:27.065020 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:27.064984 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" Nov 03 22:44:27.065020 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:27.065004 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n" event={"ID":"0da90ec4-3e54-4bfc-984b-f3d60baa79ea","Type":"ContainerDied","Data":"6243714042f12443f113025713a21980511704942cdd42f36684f5b7cd5676ac"} Nov 03 22:44:27.065303 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:27.065097 5084 scope.go:122] "RemoveContainer" containerID="1566de9592be6f7a313bc22b5db3c7430b1d3a16f731b933bfdab41dd15d3177" Nov 03 22:44:27.242366 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:27.242325 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n"] Nov 03 22:44:27.294057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:27.294015 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a9ef23-ccdd44d88-sg94n"] Nov 03 22:44:28.636564 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:28.636521 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da90ec4-3e54-4bfc-984b-f3d60baa79ea" path="/var/lib/kubelet/pods/0da90ec4-3e54-4bfc-984b-f3d60baa79ea/volumes" Nov 03 22:44:30.388257 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.388227 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:44:30.402367 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.402317 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-podinfo\") pod \"2dc0836c-8215-4302-ac4d-67da5befb2a6\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " Nov 03 22:44:30.402588 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.402564 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc0836c-8215-4302-ac4d-67da5befb2a6-kube-api-access-xtfm7\" (UniqueName: \"kubernetes.io/projected/2dc0836c-8215-4302-ac4d-67da5befb2a6-kube-api-access-xtfm7\") pod \"2dc0836c-8215-4302-ac4d-67da5befb2a6\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " Nov 03 22:44:30.402676 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.402657 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-workload-certs\") pod \"2dc0836c-8215-4302-ac4d-67da5befb2a6\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " Nov 03 22:44:30.402745 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.402715 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-credential-socket\") pod \"2dc0836c-8215-4302-ac4d-67da5befb2a6\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " Nov 03 22:44:30.402786 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.402763 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-data\") pod \"2dc0836c-8215-4302-ac4d-67da5befb2a6\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " Nov 03 22:44:30.402845 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.402819 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-envoy\") pod \"2dc0836c-8215-4302-ac4d-67da5befb2a6\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " Nov 03 22:44:30.402933 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.402916 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-token\" (UniqueName: \"kubernetes.io/projected/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-token\") pod \"2dc0836c-8215-4302-ac4d-67da5befb2a6\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " Nov 03 22:44:30.402998 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.402981 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dc0836c-8215-4302-ac4d-67da5befb2a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc0836c-8215-4302-ac4d-67da5befb2a6-istiod-ca-cert\") pod \"2dc0836c-8215-4302-ac4d-67da5befb2a6\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " Nov 03 22:44:30.403050 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.403007 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-workload-socket\") pod \"2dc0836c-8215-4302-ac4d-67da5befb2a6\" (UID: \"2dc0836c-8215-4302-ac4d-67da5befb2a6\") " Nov 03 22:44:30.403050 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.403029 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-credential-socket" pod "2dc0836c-8215-4302-ac4d-67da5befb2a6" (UID: "2dc0836c-8215-4302-ac4d-67da5befb2a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.403689 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.403368 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-workload-certs" pod "2dc0836c-8215-4302-ac4d-67da5befb2a6" (UID: "2dc0836c-8215-4302-ac4d-67da5befb2a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.403689 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.403634 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-data" pod "2dc0836c-8215-4302-ac4d-67da5befb2a6" (UID: "2dc0836c-8215-4302-ac4d-67da5befb2a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.403868 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.403685 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc0836c-8215-4302-ac4d-67da5befb2a6-istiod-ca-cert" pod "2dc0836c-8215-4302-ac4d-67da5befb2a6" (UID: "2dc0836c-8215-4302-ac4d-67da5befb2a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:30.403868 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.403843 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-workload-socket" pod "2dc0836c-8215-4302-ac4d-67da5befb2a6" (UID: "2dc0836c-8215-4302-ac4d-67da5befb2a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.404051 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.404024 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc0836c-8215-4302-ac4d-67da5befb2a6-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.404249 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.404055 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.404249 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.404089 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.404249 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.404105 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.404249 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.404120 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.405476 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.405440 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-token" pod "2dc0836c-8215-4302-ac4d-67da5befb2a6" (UID: "2dc0836c-8215-4302-ac4d-67da5befb2a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.405605 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.405522 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc0836c-8215-4302-ac4d-67da5befb2a6-kube-api-access-xtfm7" pod "2dc0836c-8215-4302-ac4d-67da5befb2a6" (UID: "2dc0836c-8215-4302-ac4d-67da5befb2a6"). InnerVolumeSpecName "kube-api-access-xtfm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.405653 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.405627 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-envoy" pod "2dc0836c-8215-4302-ac4d-67da5befb2a6" (UID: "2dc0836c-8215-4302-ac4d-67da5befb2a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.405876 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.405853 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-podinfo" pod "2dc0836c-8215-4302-ac4d-67da5befb2a6" (UID: "2dc0836c-8215-4302-ac4d-67da5befb2a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:44:30.431369 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.431331 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:44:30.504984 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.504943 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-podinfo\") pod \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " Nov 03 22:44:30.505217 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.505028 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-kube-api-access-cq9zh\" (UniqueName: \"kubernetes.io/projected/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-kube-api-access-cq9zh\") pod \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " Nov 03 22:44:30.505217 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.505145 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-workload-certs\") pod \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " Nov 03 22:44:30.505217 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.505200 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-envoy\") pod \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " Nov 03 22:44:30.505388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.505252 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-token\" (UniqueName: \"kubernetes.io/projected/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-token\") pod \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " Nov 03 22:44:30.505388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.505279 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-credential-socket\") pod \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " Nov 03 22:44:30.505388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.505331 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istiod-ca-cert\") pod \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " Nov 03 22:44:30.505555 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.505411 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-workload-socket\") pod \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " Nov 03 22:44:30.505555 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.505454 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-workload-certs" pod "ad18cacf-d8cc-4b0d-909e-7c60805abe2f" (UID: "ad18cacf-d8cc-4b0d-909e-7c60805abe2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.505555 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.505471 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-data\") pod \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\" (UID: \"ad18cacf-d8cc-4b0d-909e-7c60805abe2f\") " Nov 03 22:44:30.505745 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.505720 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-workload-socket" pod "ad18cacf-d8cc-4b0d-909e-7c60805abe2f" (UID: "ad18cacf-d8cc-4b0d-909e-7c60805abe2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.506244 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.505865 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istiod-ca-cert" pod "ad18cacf-d8cc-4b0d-909e-7c60805abe2f" (UID: "ad18cacf-d8cc-4b0d-909e-7c60805abe2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:30.506244 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.505968 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-data" pod "ad18cacf-d8cc-4b0d-909e-7c60805abe2f" (UID: "ad18cacf-d8cc-4b0d-909e-7c60805abe2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.506244 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.506200 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-credential-socket" pod "ad18cacf-d8cc-4b0d-909e-7c60805abe2f" (UID: "ad18cacf-d8cc-4b0d-909e-7c60805abe2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.506506 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.506282 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.506506 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.506309 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.506506 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.506325 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.506506 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.506339 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.506506 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.506354 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.506506 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.506370 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc0836c-8215-4302-ac4d-67da5befb2a6-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.506506 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.506384 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.506506 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.506398 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtfm7\" (UniqueName: \"kubernetes.io/projected/2dc0836c-8215-4302-ac4d-67da5befb2a6-kube-api-access-xtfm7\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.507560 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.507536 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-envoy" pod "ad18cacf-d8cc-4b0d-909e-7c60805abe2f" (UID: "ad18cacf-d8cc-4b0d-909e-7c60805abe2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:30.507644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.507611 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-podinfo" pod "ad18cacf-d8cc-4b0d-909e-7c60805abe2f" (UID: "ad18cacf-d8cc-4b0d-909e-7c60805abe2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:44:30.507644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.507628 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-kube-api-access-cq9zh" pod "ad18cacf-d8cc-4b0d-909e-7c60805abe2f" (UID: "ad18cacf-d8cc-4b0d-909e-7c60805abe2f"). InnerVolumeSpecName "kube-api-access-cq9zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.507782 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.507767 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-token" pod "ad18cacf-d8cc-4b0d-909e-7c60805abe2f" (UID: "ad18cacf-d8cc-4b0d-909e-7c60805abe2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.608340 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.608245 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.608340 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.608280 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.608340 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.608295 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.608340 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.608307 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:30.608340 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:30.608318 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq9zh\" (UniqueName: \"kubernetes.io/projected/ad18cacf-d8cc-4b0d-909e-7c60805abe2f-kube-api-access-cq9zh\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:44:31.106554 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.106484 5084 generic.go:356] "Generic (PLEG): container finished" podID="2dc0836c-8215-4302-ac4d-67da5befb2a6" containerID="309de7a6c5d1b32b213b2c10a24e21a26e98d13bf5207a841888df6fcf849b22" exitCode=0 Nov 03 22:44:31.106728 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.106556 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" Nov 03 22:44:31.106728 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.106654 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" event={"ID":"2dc0836c-8215-4302-ac4d-67da5befb2a6","Type":"ContainerDied","Data":"309de7a6c5d1b32b213b2c10a24e21a26e98d13bf5207a841888df6fcf849b22"} Nov 03 22:44:31.106728 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.106711 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk" event={"ID":"2dc0836c-8215-4302-ac4d-67da5befb2a6","Type":"ContainerDied","Data":"496142a220eee3808cac8527b4f86c5bdcfdd1e9fc42eed1b56c7aa3f7eb515d"} Nov 03 22:44:31.106886 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.106742 5084 scope.go:122] "RemoveContainer" containerID="309de7a6c5d1b32b213b2c10a24e21a26e98d13bf5207a841888df6fcf849b22" Nov 03 22:44:31.110808 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.110775 5084 generic.go:356] "Generic (PLEG): container finished" podID="ad18cacf-d8cc-4b0d-909e-7c60805abe2f" containerID="c109be0fb54c3468b9277ae28aedd0104fb9e2458f30cce82c2f32d1932edd3c" exitCode=0 Nov 03 22:44:31.110808 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.110844 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" event={"ID":"ad18cacf-d8cc-4b0d-909e-7c60805abe2f","Type":"ContainerDied","Data":"c109be0fb54c3468b9277ae28aedd0104fb9e2458f30cce82c2f32d1932edd3c"} Nov 03 22:44:31.110808 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.110924 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" event={"ID":"ad18cacf-d8cc-4b0d-909e-7c60805abe2f","Type":"ContainerDied","Data":"a40607bc4691913b340f491fa2f80df54104f5c63342c159a332eb75eae3dd44"} Nov 03 22:44:31.111407 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.111376 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z" Nov 03 22:44:31.121953 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.121923 5084 scope.go:122] "RemoveContainer" containerID="309de7a6c5d1b32b213b2c10a24e21a26e98d13bf5207a841888df6fcf849b22" Nov 03 22:44:31.122346 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:44:31.122320 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"309de7a6c5d1b32b213b2c10a24e21a26e98d13bf5207a841888df6fcf849b22\": container with ID starting with 309de7a6c5d1b32b213b2c10a24e21a26e98d13bf5207a841888df6fcf849b22 not found: ID does not exist" containerID="309de7a6c5d1b32b213b2c10a24e21a26e98d13bf5207a841888df6fcf849b22" Nov 03 22:44:31.122464 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.122355 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"309de7a6c5d1b32b213b2c10a24e21a26e98d13bf5207a841888df6fcf849b22"} err="failed to get container status \"309de7a6c5d1b32b213b2c10a24e21a26e98d13bf5207a841888df6fcf849b22\": rpc error: code = NotFound desc = could not find container \"309de7a6c5d1b32b213b2c10a24e21a26e98d13bf5207a841888df6fcf849b22\": container with ID starting with 309de7a6c5d1b32b213b2c10a24e21a26e98d13bf5207a841888df6fcf849b22 not found: ID does not exist" Nov 03 22:44:31.122464 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.122379 5084 scope.go:122] "RemoveContainer" containerID="c109be0fb54c3468b9277ae28aedd0104fb9e2458f30cce82c2f32d1932edd3c" Nov 03 22:44:31.133665 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.133627 5084 scope.go:122] "RemoveContainer" containerID="882ba25e5635fb50655d1c4f91fda96eae73ffe1b16a2731aaccc522a217aa93" Nov 03 22:44:31.148390 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.148361 5084 scope.go:122] "RemoveContainer" containerID="c109be0fb54c3468b9277ae28aedd0104fb9e2458f30cce82c2f32d1932edd3c" Nov 03 22:44:31.148780 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:44:31.148757 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c109be0fb54c3468b9277ae28aedd0104fb9e2458f30cce82c2f32d1932edd3c\": container with ID starting with c109be0fb54c3468b9277ae28aedd0104fb9e2458f30cce82c2f32d1932edd3c not found: ID does not exist" containerID="c109be0fb54c3468b9277ae28aedd0104fb9e2458f30cce82c2f32d1932edd3c" Nov 03 22:44:31.148869 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.148797 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c109be0fb54c3468b9277ae28aedd0104fb9e2458f30cce82c2f32d1932edd3c"} err="failed to get container status \"c109be0fb54c3468b9277ae28aedd0104fb9e2458f30cce82c2f32d1932edd3c\": rpc error: code = NotFound desc = could not find container \"c109be0fb54c3468b9277ae28aedd0104fb9e2458f30cce82c2f32d1932edd3c\": container with ID starting with c109be0fb54c3468b9277ae28aedd0104fb9e2458f30cce82c2f32d1932edd3c not found: ID does not exist" Nov 03 22:44:31.148869 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.148823 5084 scope.go:122] "RemoveContainer" containerID="882ba25e5635fb50655d1c4f91fda96eae73ffe1b16a2731aaccc522a217aa93" Nov 03 22:44:31.149203 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:44:31.149180 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"882ba25e5635fb50655d1c4f91fda96eae73ffe1b16a2731aaccc522a217aa93\": container with ID starting with 882ba25e5635fb50655d1c4f91fda96eae73ffe1b16a2731aaccc522a217aa93 not found: ID does not exist" containerID="882ba25e5635fb50655d1c4f91fda96eae73ffe1b16a2731aaccc522a217aa93" Nov 03 22:44:31.149338 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.149213 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"882ba25e5635fb50655d1c4f91fda96eae73ffe1b16a2731aaccc522a217aa93"} err="failed to get container status \"882ba25e5635fb50655d1c4f91fda96eae73ffe1b16a2731aaccc522a217aa93\": rpc error: code = NotFound desc = could not find container \"882ba25e5635fb50655d1c4f91fda96eae73ffe1b16a2731aaccc522a217aa93\": container with ID starting with 882ba25e5635fb50655d1c4f91fda96eae73ffe1b16a2731aaccc522a217aa93 not found: ID does not exist" Nov 03 22:44:31.279763 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.279715 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z"] Nov 03 22:44:31.285783 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.285730 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-kzg8z"] Nov 03 22:44:31.395694 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.395587 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk"] Nov 03 22:44:31.399539 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:31.399502 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bc88eb47-openshift-default-bdb864665-5mrnk"] Nov 03 22:44:32.631356 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:32.631319 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc0836c-8215-4302-ac4d-67da5befb2a6" path="/var/lib/kubelet/pods/2dc0836c-8215-4302-ac4d-67da5befb2a6/volumes" Nov 03 22:44:32.631897 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:32.631882 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad18cacf-d8cc-4b0d-909e-7c60805abe2f" path="/var/lib/kubelet/pods/ad18cacf-d8cc-4b0d-909e-7c60805abe2f/volumes" Nov 03 22:44:42.023700 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:44:42.023660 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:44:47.767705 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:47.767660 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5"] Nov 03 22:44:56.486089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:56.445499 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:56.502086 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:56.490335 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" podUID="8cebf3c1-c5fb-4feb-beac-7cf2e8a01134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:44:57.236641 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:57.236245 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-l96p9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:44:57.236641 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:44:57.236340 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" podUID="d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:00.206407 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.206359 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7"] Nov 03 22:45:00.207088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.207056 5084 cpu_manager.go:414] "containerMap: removing container" podUID="ad18cacf-d8cc-4b0d-909e-7c60805abe2f" containerName="istio-proxy" Nov 03 22:45:00.207142 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.207089 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad18cacf-d8cc-4b0d-909e-7c60805abe2f" containerName="istio-proxy" Nov 03 22:45:00.207142 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.207107 5084 cpu_manager.go:414] "containerMap: removing container" podUID="0da90ec4-3e54-4bfc-984b-f3d60baa79ea" containerName="maas-api" Nov 03 22:45:00.207142 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.207112 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da90ec4-3e54-4bfc-984b-f3d60baa79ea" containerName="maas-api" Nov 03 22:45:00.207245 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.207152 5084 cpu_manager.go:414] "containerMap: removing container" podUID="2dc0836c-8215-4302-ac4d-67da5befb2a6" containerName="istio-proxy" Nov 03 22:45:00.207245 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.207160 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc0836c-8215-4302-ac4d-67da5befb2a6" containerName="istio-proxy" Nov 03 22:45:00.207310 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.207284 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad18cacf-d8cc-4b0d-909e-7c60805abe2f" containerName="istio-proxy" Nov 03 22:45:00.207310 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.207294 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da90ec4-3e54-4bfc-984b-f3d60baa79ea" containerName="maas-api" Nov 03 22:45:00.207310 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.207307 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc0836c-8215-4302-ac4d-67da5befb2a6" containerName="istio-proxy" Nov 03 22:45:00.212644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.212610 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx"] Nov 03 22:45:00.212816 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.212740 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" Nov 03 22:45:00.213453 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.213428 5084 cpu_manager.go:414] "containerMap: removing container" podUID="ad18cacf-d8cc-4b0d-909e-7c60805abe2f" containerName="istio-proxy" Nov 03 22:45:00.213453 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.213448 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad18cacf-d8cc-4b0d-909e-7c60805abe2f" containerName="istio-proxy" Nov 03 22:45:00.213661 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.213646 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad18cacf-d8cc-4b0d-909e-7c60805abe2f" containerName="istio-proxy" Nov 03 22:45:00.217637 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.217611 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" Nov 03 22:45:00.234860 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.234827 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f7958e-6821-48a6-9dc8-07e1632aefd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7\" (UID: \"00f7958e-6821-48a6-9dc8-07e1632aefd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" Nov 03 22:45:00.235014 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.235004 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9m5p\" (UniqueName: \"kubernetes.io/projected/00f7958e-6821-48a6-9dc8-07e1632aefd1-kube-api-access-n9m5p\") pod \"maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7\" (UID: \"00f7958e-6821-48a6-9dc8-07e1632aefd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" Nov 03 22:45:00.329914 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.329876 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:45:00.336682 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.336647 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f7958e-6821-48a6-9dc8-07e1632aefd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7\" (UID: \"00f7958e-6821-48a6-9dc8-07e1632aefd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" Nov 03 22:45:00.336891 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.336778 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9m5p\" (UniqueName: \"kubernetes.io/projected/00f7958e-6821-48a6-9dc8-07e1632aefd1-kube-api-access-n9m5p\") pod \"maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7\" (UID: \"00f7958e-6821-48a6-9dc8-07e1632aefd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" Nov 03 22:45:00.336891 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.336821 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5021ca83-4c67-41e7-8c86-ecfaf82dd04f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx\" (UID: \"5021ca83-4c67-41e7-8c86-ecfaf82dd04f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" Nov 03 22:45:00.337117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.336927 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdcvg\" (UniqueName: \"kubernetes.io/projected/5021ca83-4c67-41e7-8c86-ecfaf82dd04f-kube-api-access-jdcvg\") pod \"maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx\" (UID: \"5021ca83-4c67-41e7-8c86-ecfaf82dd04f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" Nov 03 22:45:00.337394 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.337372 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f7958e-6821-48a6-9dc8-07e1632aefd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7\" (UID: \"00f7958e-6821-48a6-9dc8-07e1632aefd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" Nov 03 22:45:00.394807 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.394773 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx"] Nov 03 22:45:00.395940 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.395876 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9m5p\" (UniqueName: \"kubernetes.io/projected/00f7958e-6821-48a6-9dc8-07e1632aefd1-kube-api-access-n9m5p\") pod \"maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7\" (UID: \"00f7958e-6821-48a6-9dc8-07e1632aefd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" Nov 03 22:45:00.416884 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.416840 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7"] Nov 03 22:45:00.438455 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.438420 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdcvg\" (UniqueName: \"kubernetes.io/projected/5021ca83-4c67-41e7-8c86-ecfaf82dd04f-kube-api-access-jdcvg\") pod \"maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx\" (UID: \"5021ca83-4c67-41e7-8c86-ecfaf82dd04f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" Nov 03 22:45:00.438663 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.438641 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5021ca83-4c67-41e7-8c86-ecfaf82dd04f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx\" (UID: \"5021ca83-4c67-41e7-8c86-ecfaf82dd04f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" Nov 03 22:45:00.439419 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.439398 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5021ca83-4c67-41e7-8c86-ecfaf82dd04f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx\" (UID: \"5021ca83-4c67-41e7-8c86-ecfaf82dd04f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" Nov 03 22:45:00.514666 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.514640 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdcvg\" (UniqueName: \"kubernetes.io/projected/5021ca83-4c67-41e7-8c86-ecfaf82dd04f-kube-api-access-jdcvg\") pod \"maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx\" (UID: \"5021ca83-4c67-41e7-8c86-ecfaf82dd04f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" Nov 03 22:45:00.530017 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.529979 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" Nov 03 22:45:00.537618 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.537584 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" Nov 03 22:45:00.937630 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.937585 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7"] Nov 03 22:45:00.938265 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:45:00.938239 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f7958e_6821_48a6_9dc8_07e1632aefd1.slice/crio-32bd660140cf94affea74d0a6580b59e2af4262afccb13f46c8d9edcb7b64f84 WatchSource:0}: Error finding container 32bd660140cf94affea74d0a6580b59e2af4262afccb13f46c8d9edcb7b64f84: Status 404 returned error can't find the container with id 32bd660140cf94affea74d0a6580b59e2af4262afccb13f46c8d9edcb7b64f84 Nov 03 22:45:00.990459 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:45:00.990266 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5021ca83_4c67_41e7_8c86_ecfaf82dd04f.slice/crio-e17d116d3b92b98844f585084fee373b8a884a96619a1f4b4419a92aa0a100db WatchSource:0}: Error finding container e17d116d3b92b98844f585084fee373b8a884a96619a1f4b4419a92aa0a100db: Status 404 returned error can't find the container with id e17d116d3b92b98844f585084fee373b8a884a96619a1f4b4419a92aa0a100db Nov 03 22:45:00.994871 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:00.994835 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx"] Nov 03 22:45:01.692647 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:01.691864 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" event={"ID":"5021ca83-4c67-41e7-8c86-ecfaf82dd04f","Type":"ContainerStarted","Data":"6a998906e645e9c2b7950c21d9a9aec0680b0ec9789d4a634231086861334415"} Nov 03 22:45:01.692647 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:01.691937 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" event={"ID":"5021ca83-4c67-41e7-8c86-ecfaf82dd04f","Type":"ContainerStarted","Data":"e17d116d3b92b98844f585084fee373b8a884a96619a1f4b4419a92aa0a100db"} Nov 03 22:45:01.718412 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:01.717429 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" event={"ID":"00f7958e-6821-48a6-9dc8-07e1632aefd1","Type":"ContainerStarted","Data":"156c25466db2597a38db2a4b85d06d01c4f923f0862f4b8eaa7b61d797742ea8"} Nov 03 22:45:01.718412 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:01.717507 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" event={"ID":"00f7958e-6821-48a6-9dc8-07e1632aefd1","Type":"ContainerStarted","Data":"32bd660140cf94affea74d0a6580b59e2af4262afccb13f46c8d9edcb7b64f84"} Nov 03 22:45:01.840462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:01.822888 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" podStartSLOduration=1.822864686 podStartE2EDuration="1.822864686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:01.816701733 +0000 UTC m=+2269.410733709" watchObservedRunningTime="2026-11-03 22:45:01.822864686 +0000 UTC m=+2269.416896661" Nov 03 22:45:01.934573 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:01.934498 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" podStartSLOduration=1.934477268 podStartE2EDuration="1.934477268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:01.931967409 +0000 UTC m=+2269.525999384" watchObservedRunningTime="2026-11-03 22:45:01.934477268 +0000 UTC m=+2269.528509248" Nov 03 22:45:02.728234 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:02.728199 5084 generic.go:356] "Generic (PLEG): container finished" podID="00f7958e-6821-48a6-9dc8-07e1632aefd1" containerID="156c25466db2597a38db2a4b85d06d01c4f923f0862f4b8eaa7b61d797742ea8" exitCode=60 Nov 03 22:45:02.728764 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:02.728285 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" event={"ID":"00f7958e-6821-48a6-9dc8-07e1632aefd1","Type":"ContainerDied","Data":"156c25466db2597a38db2a4b85d06d01c4f923f0862f4b8eaa7b61d797742ea8"} Nov 03 22:45:02.728764 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:02.728632 5084 scope.go:122] "RemoveContainer" containerID="156c25466db2597a38db2a4b85d06d01c4f923f0862f4b8eaa7b61d797742ea8" Nov 03 22:45:02.730287 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:02.730264 5084 generic.go:356] "Generic (PLEG): container finished" podID="5021ca83-4c67-41e7-8c86-ecfaf82dd04f" containerID="6a998906e645e9c2b7950c21d9a9aec0680b0ec9789d4a634231086861334415" exitCode=60 Nov 03 22:45:02.730407 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:02.730348 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" event={"ID":"5021ca83-4c67-41e7-8c86-ecfaf82dd04f","Type":"ContainerDied","Data":"6a998906e645e9c2b7950c21d9a9aec0680b0ec9789d4a634231086861334415"} Nov 03 22:45:02.730699 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:02.730574 5084 scope.go:122] "RemoveContainer" containerID="6a998906e645e9c2b7950c21d9a9aec0680b0ec9789d4a634231086861334415" Nov 03 22:45:02.791167 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:02.791122 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" podUID="3b3927af-60a2-4432-99d6-b999a96f315f" containerName="main" containerID="cri-o://1c02700858b888b781d85e71eae50a2b68567fc2b9d0cc2837fc06b4c32e3dd8" gracePeriod=45 Nov 03 22:45:03.163046 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.162997 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:45:03.288155 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.288108 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b3927af-60a2-4432-99d6-b999a96f315f-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3927af-60a2-4432-99d6-b999a96f315f-tls-certs\") pod \"3b3927af-60a2-4432-99d6-b999a96f315f\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " Nov 03 22:45:03.288345 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.288264 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-model-cache\") pod \"3b3927af-60a2-4432-99d6-b999a96f315f\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " Nov 03 22:45:03.288407 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.288379 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3927af-60a2-4432-99d6-b999a96f315f-kube-api-access-7s2hj\" (UniqueName: \"kubernetes.io/projected/3b3927af-60a2-4432-99d6-b999a96f315f-kube-api-access-7s2hj\") pod \"3b3927af-60a2-4432-99d6-b999a96f315f\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " Nov 03 22:45:03.288487 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.288469 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-dshm\") pod \"3b3927af-60a2-4432-99d6-b999a96f315f\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " Nov 03 22:45:03.288554 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.288504 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-home\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-home\") pod \"3b3927af-60a2-4432-99d6-b999a96f315f\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " Nov 03 22:45:03.288590 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.288577 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-tmp-dir\") pod \"3b3927af-60a2-4432-99d6-b999a96f315f\" (UID: \"3b3927af-60a2-4432-99d6-b999a96f315f\") " Nov 03 22:45:03.289748 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.289514 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-home" pod "3b3927af-60a2-4432-99d6-b999a96f315f" (UID: "3b3927af-60a2-4432-99d6-b999a96f315f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:03.289894 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.289759 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-tmp-dir" pod "3b3927af-60a2-4432-99d6-b999a96f315f" (UID: "3b3927af-60a2-4432-99d6-b999a96f315f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:03.290036 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.290000 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-model-cache" pod "3b3927af-60a2-4432-99d6-b999a96f315f" (UID: "3b3927af-60a2-4432-99d6-b999a96f315f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:03.292994 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.292960 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b3927af-60a2-4432-99d6-b999a96f315f-tls-certs" pod "3b3927af-60a2-4432-99d6-b999a96f315f" (UID: "3b3927af-60a2-4432-99d6-b999a96f315f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:45:03.293373 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.293335 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-dshm" pod "3b3927af-60a2-4432-99d6-b999a96f315f" (UID: "3b3927af-60a2-4432-99d6-b999a96f315f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:03.293795 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.293766 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3927af-60a2-4432-99d6-b999a96f315f-kube-api-access-7s2hj" pod "3b3927af-60a2-4432-99d6-b999a96f315f" (UID: "3b3927af-60a2-4432-99d6-b999a96f315f"). InnerVolumeSpecName "kube-api-access-7s2hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:03.390851 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.390799 5084 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-dshm\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.390851 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.390851 5084 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-home\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.391154 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.390865 5084 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-tmp-dir\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.391154 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.390878 5084 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3927af-60a2-4432-99d6-b999a96f315f-tls-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.391154 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.390891 5084 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3927af-60a2-4432-99d6-b999a96f315f-model-cache\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.391154 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.390905 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s2hj\" (UniqueName: \"kubernetes.io/projected/3b3927af-60a2-4432-99d6-b999a96f315f-kube-api-access-7s2hj\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:03.751599 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.751495 5084 generic.go:356] "Generic (PLEG): container finished" podID="3b3927af-60a2-4432-99d6-b999a96f315f" containerID="1c02700858b888b781d85e71eae50a2b68567fc2b9d0cc2837fc06b4c32e3dd8" exitCode=0 Nov 03 22:45:03.751599 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.751565 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" Nov 03 22:45:03.751599 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.751576 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" event={"ID":"3b3927af-60a2-4432-99d6-b999a96f315f","Type":"ContainerDied","Data":"1c02700858b888b781d85e71eae50a2b68567fc2b9d0cc2837fc06b4c32e3dd8"} Nov 03 22:45:03.752275 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.751624 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5" event={"ID":"3b3927af-60a2-4432-99d6-b999a96f315f","Type":"ContainerDied","Data":"bb7bd871465d856415dd0b976be6d0f13aac5e1d4f726640e0788fab982cae20"} Nov 03 22:45:03.752275 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.751642 5084 scope.go:122] "RemoveContainer" containerID="1c02700858b888b781d85e71eae50a2b68567fc2b9d0cc2837fc06b4c32e3dd8" Nov 03 22:45:03.755346 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.755215 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" event={"ID":"5021ca83-4c67-41e7-8c86-ecfaf82dd04f","Type":"ContainerStarted","Data":"88d63fb3f259e87a715fe584fde7892b4a17b162e369a7be542a695624c19487"} Nov 03 22:45:03.762329 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.762289 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" event={"ID":"00f7958e-6821-48a6-9dc8-07e1632aefd1","Type":"ContainerStarted","Data":"3da44e3d2d9d33e9af2a492e961e7ace28e57a00c0b97e2ef69bd156b304ca11"} Nov 03 22:45:03.792422 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.792387 5084 scope.go:122] "RemoveContainer" containerID="1c02700858b888b781d85e71eae50a2b68567fc2b9d0cc2837fc06b4c32e3dd8" Nov 03 22:45:03.792861 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:45:03.792831 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c02700858b888b781d85e71eae50a2b68567fc2b9d0cc2837fc06b4c32e3dd8\": container with ID starting with 1c02700858b888b781d85e71eae50a2b68567fc2b9d0cc2837fc06b4c32e3dd8 not found: ID does not exist" containerID="1c02700858b888b781d85e71eae50a2b68567fc2b9d0cc2837fc06b4c32e3dd8" Nov 03 22:45:03.792930 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:03.792877 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c02700858b888b781d85e71eae50a2b68567fc2b9d0cc2837fc06b4c32e3dd8"} err="failed to get container status \"1c02700858b888b781d85e71eae50a2b68567fc2b9d0cc2837fc06b4c32e3dd8\": rpc error: code = NotFound desc = could not find container \"1c02700858b888b781d85e71eae50a2b68567fc2b9d0cc2837fc06b4c32e3dd8\": container with ID starting with 1c02700858b888b781d85e71eae50a2b68567fc2b9d0cc2837fc06b4c32e3dd8 not found: ID does not exist" Nov 03 22:45:04.036856 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:04.036819 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5"] Nov 03 22:45:04.040700 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:04.040663 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-3378fd/distinct-w1-3378fd-kserve-77b88d77dc-2jxr5"] Nov 03 22:45:04.630185 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:04.630146 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b3927af-60a2-4432-99d6-b999a96f315f" path="/var/lib/kubelet/pods/3b3927af-60a2-4432-99d6-b999a96f315f/volumes" Nov 03 22:45:04.773629 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:04.773590 5084 generic.go:356] "Generic (PLEG): container finished" podID="5021ca83-4c67-41e7-8c86-ecfaf82dd04f" containerID="88d63fb3f259e87a715fe584fde7892b4a17b162e369a7be542a695624c19487" exitCode=60 Nov 03 22:45:04.774139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:04.773663 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" event={"ID":"5021ca83-4c67-41e7-8c86-ecfaf82dd04f","Type":"ContainerDied","Data":"88d63fb3f259e87a715fe584fde7892b4a17b162e369a7be542a695624c19487"} Nov 03 22:45:04.774139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:04.773795 5084 scope.go:122] "RemoveContainer" containerID="6a998906e645e9c2b7950c21d9a9aec0680b0ec9789d4a634231086861334415" Nov 03 22:45:04.774288 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:04.774169 5084 scope.go:122] "RemoveContainer" containerID="88d63fb3f259e87a715fe584fde7892b4a17b162e369a7be542a695624c19487" Nov 03 22:45:04.774562 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:45:04.774534 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx_odh-ai-gateway-infra(5021ca83-4c67-41e7-8c86-ecfaf82dd04f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" podUID="5021ca83-4c67-41e7-8c86-ecfaf82dd04f" Nov 03 22:45:04.776591 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:04.776564 5084 generic.go:356] "Generic (PLEG): container finished" podID="00f7958e-6821-48a6-9dc8-07e1632aefd1" containerID="3da44e3d2d9d33e9af2a492e961e7ace28e57a00c0b97e2ef69bd156b304ca11" exitCode=60 Nov 03 22:45:04.776685 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:04.776645 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" event={"ID":"00f7958e-6821-48a6-9dc8-07e1632aefd1","Type":"ContainerDied","Data":"3da44e3d2d9d33e9af2a492e961e7ace28e57a00c0b97e2ef69bd156b304ca11"} Nov 03 22:45:04.777111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:04.777095 5084 scope.go:122] "RemoveContainer" containerID="3da44e3d2d9d33e9af2a492e961e7ace28e57a00c0b97e2ef69bd156b304ca11" Nov 03 22:45:04.777428 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:45:04.777411 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7_odh-ai-gateway-infra(00f7958e-6821-48a6-9dc8-07e1632aefd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" podUID="00f7958e-6821-48a6-9dc8-07e1632aefd1" Nov 03 22:45:04.790132 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:04.790099 5084 scope.go:122] "RemoveContainer" containerID="156c25466db2597a38db2a4b85d06d01c4f923f0862f4b8eaa7b61d797742ea8" Nov 03 22:45:05.788490 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:05.788455 5084 scope.go:122] "RemoveContainer" containerID="3da44e3d2d9d33e9af2a492e961e7ace28e57a00c0b97e2ef69bd156b304ca11" Nov 03 22:45:05.788914 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:45:05.788868 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7_odh-ai-gateway-infra(00f7958e-6821-48a6-9dc8-07e1632aefd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" podUID="00f7958e-6821-48a6-9dc8-07e1632aefd1" Nov 03 22:45:05.791112 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:05.791054 5084 scope.go:122] "RemoveContainer" containerID="88d63fb3f259e87a715fe584fde7892b4a17b162e369a7be542a695624c19487" Nov 03 22:45:05.791407 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:45:05.791386 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx_odh-ai-gateway-infra(5021ca83-4c67-41e7-8c86-ecfaf82dd04f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" podUID="5021ca83-4c67-41e7-8c86-ecfaf82dd04f" Nov 03 22:45:12.027556 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:45:12.027512 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:45:15.621610 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:15.621569 5084 scope.go:122] "RemoveContainer" containerID="3da44e3d2d9d33e9af2a492e961e7ace28e57a00c0b97e2ef69bd156b304ca11" Nov 03 22:45:16.879319 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:16.879228 5084 generic.go:356] "Generic (PLEG): container finished" podID="00f7958e-6821-48a6-9dc8-07e1632aefd1" containerID="3613e12d435bdf51dfdeac268458fb05ba219669a09f3aba1cf53d9a091867a2" exitCode=60 Nov 03 22:45:16.879319 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:16.879302 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" event={"ID":"00f7958e-6821-48a6-9dc8-07e1632aefd1","Type":"ContainerDied","Data":"3613e12d435bdf51dfdeac268458fb05ba219669a09f3aba1cf53d9a091867a2"} Nov 03 22:45:16.879837 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:16.879381 5084 scope.go:122] "RemoveContainer" containerID="3da44e3d2d9d33e9af2a492e961e7ace28e57a00c0b97e2ef69bd156b304ca11" Nov 03 22:45:16.880162 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:16.880141 5084 scope.go:122] "RemoveContainer" containerID="3613e12d435bdf51dfdeac268458fb05ba219669a09f3aba1cf53d9a091867a2" Nov 03 22:45:16.880572 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:45:16.880544 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7_odh-ai-gateway-infra(00f7958e-6821-48a6-9dc8-07e1632aefd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" podUID="00f7958e-6821-48a6-9dc8-07e1632aefd1" Nov 03 22:45:17.627198 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:17.624659 5084 scope.go:122] "RemoveContainer" containerID="88d63fb3f259e87a715fe584fde7892b4a17b162e369a7be542a695624c19487" Nov 03 22:45:18.010951 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:18.010909 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7"] Nov 03 22:45:18.208614 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:18.208581 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" Nov 03 22:45:18.323637 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:18.323601 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00f7958e-6821-48a6-9dc8-07e1632aefd1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f7958e-6821-48a6-9dc8-07e1632aefd1-maas-api-service-ca\") pod \"00f7958e-6821-48a6-9dc8-07e1632aefd1\" (UID: \"00f7958e-6821-48a6-9dc8-07e1632aefd1\") " Nov 03 22:45:18.323824 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:18.323660 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f7958e-6821-48a6-9dc8-07e1632aefd1-kube-api-access-n9m5p\" (UniqueName: \"kubernetes.io/projected/00f7958e-6821-48a6-9dc8-07e1632aefd1-kube-api-access-n9m5p\") pod \"00f7958e-6821-48a6-9dc8-07e1632aefd1\" (UID: \"00f7958e-6821-48a6-9dc8-07e1632aefd1\") " Nov 03 22:45:18.324132 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:18.324097 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f7958e-6821-48a6-9dc8-07e1632aefd1-maas-api-service-ca" pod "00f7958e-6821-48a6-9dc8-07e1632aefd1" (UID: "00f7958e-6821-48a6-9dc8-07e1632aefd1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:18.324312 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:18.324288 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f7958e-6821-48a6-9dc8-07e1632aefd1-maas-api-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:18.336731 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:18.326714 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f7958e-6821-48a6-9dc8-07e1632aefd1-kube-api-access-n9m5p" pod "00f7958e-6821-48a6-9dc8-07e1632aefd1" (UID: "00f7958e-6821-48a6-9dc8-07e1632aefd1"). InnerVolumeSpecName "kube-api-access-n9m5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:18.425724 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:18.425666 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9m5p\" (UniqueName: \"kubernetes.io/projected/00f7958e-6821-48a6-9dc8-07e1632aefd1-kube-api-access-n9m5p\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:18.950647 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:18.950551 5084 generic.go:356] "Generic (PLEG): container finished" podID="5021ca83-4c67-41e7-8c86-ecfaf82dd04f" containerID="7c3684fb5ee77eca9f18d6e9c7fb2a49b29f29e006024df040286b5d1eb735f2" exitCode=60 Nov 03 22:45:18.950862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:18.950639 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" event={"ID":"5021ca83-4c67-41e7-8c86-ecfaf82dd04f","Type":"ContainerDied","Data":"7c3684fb5ee77eca9f18d6e9c7fb2a49b29f29e006024df040286b5d1eb735f2"} Nov 03 22:45:18.950862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:18.950704 5084 scope.go:122] "RemoveContainer" containerID="88d63fb3f259e87a715fe584fde7892b4a17b162e369a7be542a695624c19487" Nov 03 22:45:18.951296 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:18.951267 5084 scope.go:122] "RemoveContainer" containerID="7c3684fb5ee77eca9f18d6e9c7fb2a49b29f29e006024df040286b5d1eb735f2" Nov 03 22:45:18.951725 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:45:18.951679 5084 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx_odh-ai-gateway-infra(5021ca83-4c67-41e7-8c86-ecfaf82dd04f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" podUID="5021ca83-4c67-41e7-8c86-ecfaf82dd04f" Nov 03 22:45:18.952730 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:18.952707 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" event={"ID":"00f7958e-6821-48a6-9dc8-07e1632aefd1","Type":"ContainerDied","Data":"32bd660140cf94affea74d0a6580b59e2af4262afccb13f46c8d9edcb7b64f84"} Nov 03 22:45:18.952851 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:18.952789 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7" Nov 03 22:45:18.966028 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:18.966004 5084 scope.go:122] "RemoveContainer" containerID="3613e12d435bdf51dfdeac268458fb05ba219669a09f3aba1cf53d9a091867a2" Nov 03 22:45:19.194658 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:19.194614 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7"] Nov 03 22:45:19.200605 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:19.198768 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-906673-29895765-r8vd7"] Nov 03 22:45:20.082610 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:20.082561 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx"] Nov 03 22:45:20.266498 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:20.266469 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" Nov 03 22:45:20.362760 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:20.362652 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5021ca83-4c67-41e7-8c86-ecfaf82dd04f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5021ca83-4c67-41e7-8c86-ecfaf82dd04f-maas-api-service-ca\") pod \"5021ca83-4c67-41e7-8c86-ecfaf82dd04f\" (UID: \"5021ca83-4c67-41e7-8c86-ecfaf82dd04f\") " Nov 03 22:45:20.362998 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:20.362974 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5021ca83-4c67-41e7-8c86-ecfaf82dd04f-kube-api-access-jdcvg\" (UniqueName: \"kubernetes.io/projected/5021ca83-4c67-41e7-8c86-ecfaf82dd04f-kube-api-access-jdcvg\") pod \"5021ca83-4c67-41e7-8c86-ecfaf82dd04f\" (UID: \"5021ca83-4c67-41e7-8c86-ecfaf82dd04f\") " Nov 03 22:45:20.363097 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:20.362982 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5021ca83-4c67-41e7-8c86-ecfaf82dd04f-maas-api-service-ca" pod "5021ca83-4c67-41e7-8c86-ecfaf82dd04f" (UID: "5021ca83-4c67-41e7-8c86-ecfaf82dd04f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:20.363470 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:20.363449 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5021ca83-4c67-41e7-8c86-ecfaf82dd04f-maas-api-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:20.365199 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:20.365166 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5021ca83-4c67-41e7-8c86-ecfaf82dd04f-kube-api-access-jdcvg" pod "5021ca83-4c67-41e7-8c86-ecfaf82dd04f" (UID: "5021ca83-4c67-41e7-8c86-ecfaf82dd04f"). InnerVolumeSpecName "kube-api-access-jdcvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:20.475093 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:20.474502 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdcvg\" (UniqueName: \"kubernetes.io/projected/5021ca83-4c67-41e7-8c86-ecfaf82dd04f-kube-api-access-jdcvg\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:45:20.629850 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:20.629757 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00f7958e-6821-48a6-9dc8-07e1632aefd1" path="/var/lib/kubelet/pods/00f7958e-6821-48a6-9dc8-07e1632aefd1/volumes" Nov 03 22:45:20.973276 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:20.973181 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" event={"ID":"5021ca83-4c67-41e7-8c86-ecfaf82dd04f","Type":"ContainerDied","Data":"e17d116d3b92b98844f585084fee373b8a884a96619a1f4b4419a92aa0a100db"} Nov 03 22:45:20.973276 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:20.973204 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx" Nov 03 22:45:20.973276 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:20.973251 5084 scope.go:122] "RemoveContainer" containerID="7c3684fb5ee77eca9f18d6e9c7fb2a49b29f29e006024df040286b5d1eb735f2" Nov 03 22:45:21.142793 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:21.142751 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx"] Nov 03 22:45:21.191663 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:21.191619 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b30a14-29895765-6zwkx"] Nov 03 22:45:22.630382 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:22.630298 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5021ca83-4c67-41e7-8c86-ecfaf82dd04f" path="/var/lib/kubelet/pods/5021ca83-4c67-41e7-8c86-ecfaf82dd04f/volumes" Nov 03 22:45:27.547229 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.546687 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh"] Nov 03 22:45:27.547838 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.547725 5084 cpu_manager.go:414] "containerMap: removing container" podUID="5021ca83-4c67-41e7-8c86-ecfaf82dd04f" containerName="cleanup" Nov 03 22:45:27.547838 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.547750 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5021ca83-4c67-41e7-8c86-ecfaf82dd04f" containerName="cleanup" Nov 03 22:45:27.547838 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.547788 5084 cpu_manager.go:414] "containerMap: removing container" podUID="5021ca83-4c67-41e7-8c86-ecfaf82dd04f" containerName="cleanup" Nov 03 22:45:27.547838 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.547797 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5021ca83-4c67-41e7-8c86-ecfaf82dd04f" containerName="cleanup" Nov 03 22:45:27.547838 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.547826 5084 cpu_manager.go:414] "containerMap: removing container" podUID="00f7958e-6821-48a6-9dc8-07e1632aefd1" containerName="cleanup" Nov 03 22:45:27.547838 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.547835 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f7958e-6821-48a6-9dc8-07e1632aefd1" containerName="cleanup" Nov 03 22:45:27.548184 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.547858 5084 cpu_manager.go:414] "containerMap: removing container" podUID="00f7958e-6821-48a6-9dc8-07e1632aefd1" containerName="cleanup" Nov 03 22:45:27.548184 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.547865 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f7958e-6821-48a6-9dc8-07e1632aefd1" containerName="cleanup" Nov 03 22:45:27.548184 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.547886 5084 cpu_manager.go:414] "containerMap: removing container" podUID="3b3927af-60a2-4432-99d6-b999a96f315f" containerName="main" Nov 03 22:45:27.548184 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.547893 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b3927af-60a2-4432-99d6-b999a96f315f" containerName="main" Nov 03 22:45:27.559090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.555697 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="5021ca83-4c67-41e7-8c86-ecfaf82dd04f" containerName="cleanup" Nov 03 22:45:27.559090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.555746 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f7958e-6821-48a6-9dc8-07e1632aefd1" containerName="cleanup" Nov 03 22:45:27.559090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.555764 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="5021ca83-4c67-41e7-8c86-ecfaf82dd04f" containerName="cleanup" Nov 03 22:45:27.559090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.555781 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f7958e-6821-48a6-9dc8-07e1632aefd1" containerName="cleanup" Nov 03 22:45:27.559090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.555792 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b3927af-60a2-4432-99d6-b999a96f315f" containerName="main" Nov 03 22:45:27.559090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.555805 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f7958e-6821-48a6-9dc8-07e1632aefd1" containerName="cleanup" Nov 03 22:45:27.587215 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.587172 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh"] Nov 03 22:45:27.587451 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.587310 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh" Nov 03 22:45:27.635559 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.635500 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mprf8\" (UniqueName: \"kubernetes.io/projected/3956759d-6df8-4ae5-b9db-80933d555c9c-kube-api-access-mprf8\") pod \"payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh\" (UID: \"3956759d-6df8-4ae5-b9db-80933d555c9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh" Nov 03 22:45:27.635803 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.635708 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3956759d-6df8-4ae5-b9db-80933d555c9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh\" (UID: \"3956759d-6df8-4ae5-b9db-80933d555c9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh" Nov 03 22:45:27.647312 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.647260 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7bcd0b\"" Nov 03 22:45:27.648822 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.647545 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7bcd0b-dockercfg-nsc7x\"" Nov 03 22:45:27.753123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.741050 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mprf8\" (UniqueName: \"kubernetes.io/projected/3956759d-6df8-4ae5-b9db-80933d555c9c-kube-api-access-mprf8\") pod \"payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh\" (UID: \"3956759d-6df8-4ae5-b9db-80933d555c9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh" Nov 03 22:45:27.753123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.741444 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3956759d-6df8-4ae5-b9db-80933d555c9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh\" (UID: \"3956759d-6df8-4ae5-b9db-80933d555c9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh" Nov 03 22:45:27.753123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.745269 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3956759d-6df8-4ae5-b9db-80933d555c9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh\" (UID: \"3956759d-6df8-4ae5-b9db-80933d555c9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh" Nov 03 22:45:27.753123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.745627 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t"] Nov 03 22:45:27.753123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.746717 5084 cpu_manager.go:414] "containerMap: removing container" podUID="00f7958e-6821-48a6-9dc8-07e1632aefd1" containerName="cleanup" Nov 03 22:45:27.753123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.746738 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f7958e-6821-48a6-9dc8-07e1632aefd1" containerName="cleanup" Nov 03 22:45:27.753123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.746782 5084 cpu_manager.go:414] "containerMap: removing container" podUID="5021ca83-4c67-41e7-8c86-ecfaf82dd04f" containerName="cleanup" Nov 03 22:45:27.753123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.746792 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5021ca83-4c67-41e7-8c86-ecfaf82dd04f" containerName="cleanup" Nov 03 22:45:27.753123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.746945 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="5021ca83-4c67-41e7-8c86-ecfaf82dd04f" containerName="cleanup" Nov 03 22:45:27.760403 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.760365 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t" Nov 03 22:45:27.770612 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.770580 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t"] Nov 03 22:45:27.832925 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.832755 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mprf8\" (UniqueName: \"kubernetes.io/projected/3956759d-6df8-4ae5-b9db-80933d555c9c-kube-api-access-mprf8\") pod \"payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh\" (UID: \"3956759d-6df8-4ae5-b9db-80933d555c9c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh" Nov 03 22:45:27.844147 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.842575 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br75g\" (UniqueName: \"kubernetes.io/projected/29c0957d-9d08-4667-b81e-fe1347e4805a-kube-api-access-br75g\") pod \"payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t\" (UID: \"29c0957d-9d08-4667-b81e-fe1347e4805a\") " pod="openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t" Nov 03 22:45:27.844147 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.842772 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c0957d-9d08-4667-b81e-fe1347e4805a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t\" (UID: \"29c0957d-9d08-4667-b81e-fe1347e4805a\") " pod="openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t" Nov 03 22:45:27.907737 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.907693 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh" Nov 03 22:45:27.960162 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.957928 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c0957d-9d08-4667-b81e-fe1347e4805a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t\" (UID: \"29c0957d-9d08-4667-b81e-fe1347e4805a\") " pod="openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t" Nov 03 22:45:27.960162 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.958164 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br75g\" (UniqueName: \"kubernetes.io/projected/29c0957d-9d08-4667-b81e-fe1347e4805a-kube-api-access-br75g\") pod \"payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t\" (UID: \"29c0957d-9d08-4667-b81e-fe1347e4805a\") " pod="openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t" Nov 03 22:45:27.960761 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:27.960730 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c0957d-9d08-4667-b81e-fe1347e4805a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t\" (UID: \"29c0957d-9d08-4667-b81e-fe1347e4805a\") " pod="openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t" Nov 03 22:45:28.019771 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:28.019731 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br75g\" (UniqueName: \"kubernetes.io/projected/29c0957d-9d08-4667-b81e-fe1347e4805a-kube-api-access-br75g\") pod \"payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t\" (UID: \"29c0957d-9d08-4667-b81e-fe1347e4805a\") " pod="openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t" Nov 03 22:45:28.142656 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:28.142559 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t" Nov 03 22:45:28.367387 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:28.367349 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh"] Nov 03 22:45:28.369539 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:45:28.369495 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3956759d_6df8_4ae5_b9db_80933d555c9c.slice/crio-f874424d482edf003200b0769de448af25fe48a016bc414f5f094bead71fd967 WatchSource:0}: Error finding container f874424d482edf003200b0769de448af25fe48a016bc414f5f094bead71fd967: Status 404 returned error can't find the container with id f874424d482edf003200b0769de448af25fe48a016bc414f5f094bead71fd967 Nov 03 22:45:28.551770 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:28.551734 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t"] Nov 03 22:45:28.553997 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:45:28.553957 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c0957d_9d08_4667_b81e_fe1347e4805a.slice/crio-4ccaee9cd8072c3c5bdfa739f5657fcc0f436617259740d19415a1a9629a3839 WatchSource:0}: Error finding container 4ccaee9cd8072c3c5bdfa739f5657fcc0f436617259740d19415a1a9629a3839: Status 404 returned error can't find the container with id 4ccaee9cd8072c3c5bdfa739f5657fcc0f436617259740d19415a1a9629a3839 Nov 03 22:45:29.151129 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:29.151007 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh" event={"ID":"3956759d-6df8-4ae5-b9db-80933d555c9c","Type":"ContainerStarted","Data":"d49d4839593c94fef10dc287faf7aa6219ddfa72964fe8a205e75155c0e27858"} Nov 03 22:45:29.151824 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:29.151790 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh" event={"ID":"3956759d-6df8-4ae5-b9db-80933d555c9c","Type":"ContainerStarted","Data":"f874424d482edf003200b0769de448af25fe48a016bc414f5f094bead71fd967"} Nov 03 22:45:29.151965 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:29.151844 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh" Nov 03 22:45:29.153928 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:29.153671 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t" event={"ID":"29c0957d-9d08-4667-b81e-fe1347e4805a","Type":"ContainerStarted","Data":"cb34aea036f918f264d75b7902d7066f5392e39168c88d962663b2a74987722a"} Nov 03 22:45:29.153928 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:29.153738 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t" event={"ID":"29c0957d-9d08-4667-b81e-fe1347e4805a","Type":"ContainerStarted","Data":"4ccaee9cd8072c3c5bdfa739f5657fcc0f436617259740d19415a1a9629a3839"} Nov 03 22:45:29.153928 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:29.153770 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t" Nov 03 22:45:29.270256 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:29.270186 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh" podStartSLOduration=2.270164175 podStartE2EDuration="2.270164175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:29.266953591 +0000 UTC m=+2296.860985566" watchObservedRunningTime="2026-11-03 22:45:29.270164175 +0000 UTC m=+2296.864196156" Nov 03 22:45:29.424122 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:29.414650 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t" podStartSLOduration=2.414622663 podStartE2EDuration="2.414622663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:29.39253538 +0000 UTC m=+2296.986567353" watchObservedRunningTime="2026-11-03 22:45:29.414622663 +0000 UTC m=+2297.008654638" Nov 03 22:45:33.040115 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.040053 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9"] Nov 03 22:45:33.045379 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.045348 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.094177 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.094119 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-home\") pod \"test-model-345da9-kserve-65d869df47-tpjq9\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.094431 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.094206 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-tmp-dir\") pod \"test-model-345da9-kserve-65d869df47-tpjq9\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.094431 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.094293 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df48770c-9c54-43d0-87c8-bc7a65bebf2c-tls-certs\") pod \"test-model-345da9-kserve-65d869df47-tpjq9\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.094431 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.094383 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-dshm\") pod \"test-model-345da9-kserve-65d869df47-tpjq9\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.094431 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.094429 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j2sx\" (UniqueName: \"kubernetes.io/projected/df48770c-9c54-43d0-87c8-bc7a65bebf2c-kube-api-access-7j2sx\") pod \"test-model-345da9-kserve-65d869df47-tpjq9\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.094842 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.094521 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-model-cache\") pod \"test-model-345da9-kserve-65d869df47-tpjq9\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.148425 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.148384 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-345da9\"/\"kube-root-ca.crt\"" Nov 03 22:45:33.149937 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.149877 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-345da9\"/\"openshift-service-ca.crt\"" Nov 03 22:45:33.150180 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.150130 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-345da9\"/\"default-dockercfg-78ktb\"" Nov 03 22:45:33.150276 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.150230 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-345da9\"/\"test-model-345da9-kserve-self-signed-certs\"" Nov 03 22:45:33.156504 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.156459 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9"] Nov 03 22:45:33.211677 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.211579 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df48770c-9c54-43d0-87c8-bc7a65bebf2c-tls-certs\") pod \"test-model-345da9-kserve-65d869df47-tpjq9\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.212623 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.212587 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-dshm\") pod \"test-model-345da9-kserve-65d869df47-tpjq9\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.213300 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.213276 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j2sx\" (UniqueName: \"kubernetes.io/projected/df48770c-9c54-43d0-87c8-bc7a65bebf2c-kube-api-access-7j2sx\") pod \"test-model-345da9-kserve-65d869df47-tpjq9\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.220714 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.220676 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-model-cache\") pod \"test-model-345da9-kserve-65d869df47-tpjq9\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.221220 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.221175 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-model-cache\") pod \"test-model-345da9-kserve-65d869df47-tpjq9\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.221611 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.221582 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-home\") pod \"test-model-345da9-kserve-65d869df47-tpjq9\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.222229 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.222197 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-tmp-dir\") pod \"test-model-345da9-kserve-65d869df47-tpjq9\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.222899 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.222877 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-tmp-dir\") pod \"test-model-345da9-kserve-65d869df47-tpjq9\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.223117 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.222084 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-home\") pod \"test-model-345da9-kserve-65d869df47-tpjq9\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.224983 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.224957 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-dshm\") pod \"test-model-345da9-kserve-65d869df47-tpjq9\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.258427 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.258391 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df48770c-9c54-43d0-87c8-bc7a65bebf2c-tls-certs\") pod \"test-model-345da9-kserve-65d869df47-tpjq9\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.303513 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.303429 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j2sx\" (UniqueName: \"kubernetes.io/projected/df48770c-9c54-43d0-87c8-bc7a65bebf2c-kube-api-access-7j2sx\") pod \"test-model-345da9-kserve-65d869df47-tpjq9\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.361721 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.361677 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:33.829521 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:33.829485 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9"] Nov 03 22:45:33.830498 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:45:33.830469 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf48770c_9c54_43d0_87c8_bc7a65bebf2c.slice/crio-2f67b8640184c2bab8634088622e73311c17e94895d7cbf2a225c91e9b78e9e2 WatchSource:0}: Error finding container 2f67b8640184c2bab8634088622e73311c17e94895d7cbf2a225c91e9b78e9e2: Status 404 returned error can't find the container with id 2f67b8640184c2bab8634088622e73311c17e94895d7cbf2a225c91e9b78e9e2 Nov 03 22:45:34.229849 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:34.229707 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" event={"ID":"df48770c-9c54-43d0-87c8-bc7a65bebf2c","Type":"ContainerStarted","Data":"6a1e70a0d8fd1d73f15f9f2001698d45c1d4ae8b13a61279c46492aee43ef6de"} Nov 03 22:45:34.229849 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:34.229794 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" event={"ID":"df48770c-9c54-43d0-87c8-bc7a65bebf2c","Type":"ContainerStarted","Data":"2f67b8640184c2bab8634088622e73311c17e94895d7cbf2a225c91e9b78e9e2"} Nov 03 22:45:34.346618 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:34.346550 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" podStartSLOduration=2.346532481 podStartE2EDuration="2.346532481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:34.344652316 +0000 UTC m=+2301.938684289" watchObservedRunningTime="2026-11-03 22:45:34.346532481 +0000 UTC m=+2301.940564454" Nov 03 22:45:34.362811 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:34.362776 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:40.167883 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:40.167847 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh" Nov 03 22:45:40.168357 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:40.167943 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t" Nov 03 22:45:42.033366 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:45:42.033318 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:45:43.372088 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:43.371838 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:43.484138 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:43.484100 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:44.373751 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:44.373703 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:45:54.842247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:54.842198 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2"] Nov 03 22:45:54.847436 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:54.847409 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:45:54.867954 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:54.867912 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w9mp\" (UniqueName: \"kubernetes.io/projected/ae836564-7d17-4919-8a65-f655fabe9759-kube-api-access-6w9mp\") pod \"maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2\" (UID: \"ae836564-7d17-4919-8a65-f655fabe9759\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:45:54.868179 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:54.867966 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae836564-7d17-4919-8a65-f655fabe9759-metrics-certs\") pod \"maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2\" (UID: \"ae836564-7d17-4919-8a65-f655fabe9759\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:45:54.868264 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:54.868243 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae836564-7d17-4919-8a65-f655fabe9759-maas-api-tls\") pod \"maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2\" (UID: \"ae836564-7d17-4919-8a65-f655fabe9759\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:45:54.868425 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:54.868396 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae836564-7d17-4919-8a65-f655fabe9759-openshift-service-ca\") pod \"maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2\" (UID: \"ae836564-7d17-4919-8a65-f655fabe9759\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:45:54.901710 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:54.901660 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2"] Nov 03 22:45:54.945655 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:54.945604 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-01bb3a-serving-cert\"" Nov 03 22:45:54.969806 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:54.969746 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w9mp\" (UniqueName: \"kubernetes.io/projected/ae836564-7d17-4919-8a65-f655fabe9759-kube-api-access-6w9mp\") pod \"maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2\" (UID: \"ae836564-7d17-4919-8a65-f655fabe9759\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:45:54.970033 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:54.969808 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae836564-7d17-4919-8a65-f655fabe9759-metrics-certs\") pod \"maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2\" (UID: \"ae836564-7d17-4919-8a65-f655fabe9759\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:45:54.970033 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:54.969871 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae836564-7d17-4919-8a65-f655fabe9759-maas-api-tls\") pod \"maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2\" (UID: \"ae836564-7d17-4919-8a65-f655fabe9759\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:45:54.970033 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:54.969958 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae836564-7d17-4919-8a65-f655fabe9759-openshift-service-ca\") pod \"maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2\" (UID: \"ae836564-7d17-4919-8a65-f655fabe9759\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:45:54.970236 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:45:54.970087 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-serving-cert: secret "maas-api-e2e-inf-b-01bb3a-serving-cert" not found Nov 03 22:45:54.970236 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:45:54.970170 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae836564-7d17-4919-8a65-f655fabe9759-maas-api-tls podName:ae836564-7d17-4919-8a65-f655fabe9759 nodeName:}" failed. No retries permitted until 2026-11-03 22:45:55.470147309 +0000 UTC m=+2323.064179264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae836564-7d17-4919-8a65-f655fabe9759-maas-api-tls") pod "maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" (UID: "ae836564-7d17-4919-8a65-f655fabe9759") : secret "maas-api-e2e-inf-b-01bb3a-serving-cert" not found Nov 03 22:45:54.970738 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:54.970712 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae836564-7d17-4919-8a65-f655fabe9759-openshift-service-ca\") pod \"maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2\" (UID: \"ae836564-7d17-4919-8a65-f655fabe9759\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:45:54.972549 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:54.972520 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae836564-7d17-4919-8a65-f655fabe9759-metrics-certs\") pod \"maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2\" (UID: \"ae836564-7d17-4919-8a65-f655fabe9759\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:45:55.028783 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:55.028749 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w9mp\" (UniqueName: \"kubernetes.io/projected/ae836564-7d17-4919-8a65-f655fabe9759-kube-api-access-6w9mp\") pod \"maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2\" (UID: \"ae836564-7d17-4919-8a65-f655fabe9759\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:45:55.487599 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:55.487552 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae836564-7d17-4919-8a65-f655fabe9759-maas-api-tls\") pod \"maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2\" (UID: \"ae836564-7d17-4919-8a65-f655fabe9759\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:45:55.490241 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:55.490215 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae836564-7d17-4919-8a65-f655fabe9759-maas-api-tls\") pod \"maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2\" (UID: \"ae836564-7d17-4919-8a65-f655fabe9759\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:45:55.767262 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:55.767207 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:45:56.160518 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:45:56.160479 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae836564_7d17_4919_8a65_f655fabe9759.slice/crio-cb177c14d74f96556f5dad48597b57e08780125e0b2c721a1be7de8f28f39f54 WatchSource:0}: Error finding container cb177c14d74f96556f5dad48597b57e08780125e0b2c721a1be7de8f28f39f54: Status 404 returned error can't find the container with id cb177c14d74f96556f5dad48597b57e08780125e0b2c721a1be7de8f28f39f54 Nov 03 22:45:56.162002 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:56.161971 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2"] Nov 03 22:45:56.635208 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:56.635170 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" event={"ID":"ae836564-7d17-4919-8a65-f655fabe9759","Type":"ContainerStarted","Data":"cb177c14d74f96556f5dad48597b57e08780125e0b2c721a1be7de8f28f39f54"} Nov 03 22:45:57.465366 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.465323 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm"] Nov 03 22:45:57.470293 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.470259 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.516361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.516316 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5188a0-6554-4eb7-8567-0613709d03d9-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.516567 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.516442 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.516567 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.516500 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.516697 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.516579 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.516697 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.516620 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w7xz\" (UniqueName: \"kubernetes.io/projected/bb5188a0-6554-4eb7-8567-0613709d03d9-kube-api-access-6w7xz\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.516822 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.516761 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.516822 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.516801 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.516932 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.516856 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.516932 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.516916 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.618601 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.618561 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5188a0-6554-4eb7-8567-0613709d03d9-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.618801 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.618723 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.618801 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.618773 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.618801 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.618796 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.618947 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.618819 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w7xz\" (UniqueName: \"kubernetes.io/projected/bb5188a0-6554-4eb7-8567-0613709d03d9-kube-api-access-6w7xz\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.618947 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.618903 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.618947 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.618938 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.619203 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.618959 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.619203 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.619027 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.619203 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.619177 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-workload-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.619349 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.619205 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-credential-socket\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.619505 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.619471 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-data\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.619505 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.619495 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-workload-certs\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.619650 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.619479 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5188a0-6554-4eb7-8567-0613709d03d9-istiod-ca-cert\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.621474 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.621402 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-envoy\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.621929 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.621881 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-podinfo\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.646377 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.646333 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" event={"ID":"ae836564-7d17-4919-8a65-f655fabe9759","Type":"ContainerStarted","Data":"f68420503c9bc5a9672d09ab7f5e77c560f61f5b014c5e88c052591a6ddc9088"} Nov 03 22:45:57.646555 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.646430 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:45:57.668741 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.668705 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm"] Nov 03 22:45:57.677777 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.677734 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-token\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.677969 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.677940 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w7xz\" (UniqueName: \"kubernetes.io/projected/bb5188a0-6554-4eb7-8567-0613709d03d9-kube-api-access-6w7xz\") pod \"e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:57.767424 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.767368 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" podStartSLOduration=2.935855092 podStartE2EDuration="3.767353321s" totalImagesPullingTime="831.498229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:57.764163512 +0000 UTC m=+2325.358195485" watchObservedRunningTime="2026-11-03 22:45:57.767353321 +0000 UTC m=+2325.361385296" Nov 03 22:45:57.786324 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:57.786276 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:58.257160 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:58.237552 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm"] Nov 03 22:45:58.351959 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:45:58.351918 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5188a0_6554_4eb7_8567_0613709d03d9.slice/crio-367869ea2bef77ce2fc88c17757024750506eb011fa8fd11af1414186f875da6 WatchSource:0}: Error finding container 367869ea2bef77ce2fc88c17757024750506eb011fa8fd11af1414186f875da6: Status 404 returned error can't find the container with id 367869ea2bef77ce2fc88c17757024750506eb011fa8fd11af1414186f875da6 Nov 03 22:45:58.375560 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:58.375506 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:45:58.375687 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:58.375611 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:45:58.375687 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:58.375652 5084 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 03 22:45:58.658492 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:58.658454 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" event={"ID":"bb5188a0-6554-4eb7-8567-0613709d03d9","Type":"ContainerStarted","Data":"fcab45c5e325a7cb1fe394921068dca1d24302e10a213fad173912e8b1a58071"} Nov 03 22:45:58.659011 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:58.658518 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" event={"ID":"bb5188a0-6554-4eb7-8567-0613709d03d9","Type":"ContainerStarted","Data":"367869ea2bef77ce2fc88c17757024750506eb011fa8fd11af1414186f875da6"} Nov 03 22:45:58.782197 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:58.782135 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" podStartSLOduration=1.782118844 podStartE2EDuration="1.782118844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:58.77821281 +0000 UTC m=+2326.372244780" watchObservedRunningTime="2026-11-03 22:45:58.782118844 +0000 UTC m=+2326.376150814" Nov 03 22:45:58.787178 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:58.787135 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:45:59.790240 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:59.790196 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 22:45:59.790751 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:45:59.790265 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" podUID="bb5188a0-6554-4eb7-8567-0613709d03d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 22:46:00.789524 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:00.789474 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 22:46:00.790742 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:00.790698 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" podUID="bb5188a0-6554-4eb7-8567-0613709d03d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 22:46:01.804692 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:01.804649 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 22:46:01.805260 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:01.804791 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" podUID="bb5188a0-6554-4eb7-8567-0613709d03d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 22:46:02.787156 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:02.787106 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 22:46:02.787380 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:02.787193 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" podUID="bb5188a0-6554-4eb7-8567-0613709d03d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 22:46:03.687449 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:03.687411 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:46:03.787991 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:03.787947 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 22:46:03.788283 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:03.788249 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" podUID="bb5188a0-6554-4eb7-8567-0613709d03d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 22:46:04.787036 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:04.786994 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 22:46:04.787543 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:04.787111 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" podUID="bb5188a0-6554-4eb7-8567-0613709d03d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 22:46:05.786629 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:05.786594 5084 patch_prober.go:29] interesting pod/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 22:46:05.786833 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:05.786655 5084 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" podUID="bb5188a0-6554-4eb7-8567-0613709d03d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 22:46:06.792045 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:06.792010 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:46:06.792548 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:06.792498 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:46:06.793638 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:06.793612 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:46:12.025862 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:46:12.025825 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:46:14.029774 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.029728 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw"] Nov 03 22:46:14.034857 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.034829 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.093135 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.092518 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-home\") pod \"test-model-b30a14-kserve-7f5bcc8d4f-79gzw\" (UID: \"49089888-27d7-482d-a915-450990214859\") " pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.093135 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.092634 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-dshm\") pod \"test-model-b30a14-kserve-7f5bcc8d4f-79gzw\" (UID: \"49089888-27d7-482d-a915-450990214859\") " pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.093135 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.092683 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-tmp-dir\") pod \"test-model-b30a14-kserve-7f5bcc8d4f-79gzw\" (UID: \"49089888-27d7-482d-a915-450990214859\") " pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.093135 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.092736 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brkrw\" (UniqueName: \"kubernetes.io/projected/49089888-27d7-482d-a915-450990214859-kube-api-access-brkrw\") pod \"test-model-b30a14-kserve-7f5bcc8d4f-79gzw\" (UID: \"49089888-27d7-482d-a915-450990214859\") " pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.093135 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.092790 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49089888-27d7-482d-a915-450990214859-tls-certs\") pod \"test-model-b30a14-kserve-7f5bcc8d4f-79gzw\" (UID: \"49089888-27d7-482d-a915-450990214859\") " pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.093135 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.092832 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-model-cache\") pod \"test-model-b30a14-kserve-7f5bcc8d4f-79gzw\" (UID: \"49089888-27d7-482d-a915-450990214859\") " pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.098931 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.098891 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw"] Nov 03 22:46:14.142765 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.142721 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b30a14\"/\"openshift-service-ca.crt\"" Nov 03 22:46:14.142962 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.142932 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b30a14\"/\"test-model-b30a14-kserve-self-signed-certs\"" Nov 03 22:46:14.143033 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.143016 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b30a14\"/\"kube-root-ca.crt\"" Nov 03 22:46:14.143124 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.143104 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b30a14\"/\"default-dockercfg-gmzgj\"" Nov 03 22:46:14.288602 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.288496 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brkrw\" (UniqueName: \"kubernetes.io/projected/49089888-27d7-482d-a915-450990214859-kube-api-access-brkrw\") pod \"test-model-b30a14-kserve-7f5bcc8d4f-79gzw\" (UID: \"49089888-27d7-482d-a915-450990214859\") " pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.290644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.290599 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49089888-27d7-482d-a915-450990214859-tls-certs\") pod \"test-model-b30a14-kserve-7f5bcc8d4f-79gzw\" (UID: \"49089888-27d7-482d-a915-450990214859\") " pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.290962 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.290937 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-model-cache\") pod \"test-model-b30a14-kserve-7f5bcc8d4f-79gzw\" (UID: \"49089888-27d7-482d-a915-450990214859\") " pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.291435 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.291371 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-home\") pod \"test-model-b30a14-kserve-7f5bcc8d4f-79gzw\" (UID: \"49089888-27d7-482d-a915-450990214859\") " pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.291691 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.291669 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-dshm\") pod \"test-model-b30a14-kserve-7f5bcc8d4f-79gzw\" (UID: \"49089888-27d7-482d-a915-450990214859\") " pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.291902 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.291883 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-tmp-dir\") pod \"test-model-b30a14-kserve-7f5bcc8d4f-79gzw\" (UID: \"49089888-27d7-482d-a915-450990214859\") " pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.293904 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.293877 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-tmp-dir\") pod \"test-model-b30a14-kserve-7f5bcc8d4f-79gzw\" (UID: \"49089888-27d7-482d-a915-450990214859\") " pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.298632 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.297407 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-home\") pod \"test-model-b30a14-kserve-7f5bcc8d4f-79gzw\" (UID: \"49089888-27d7-482d-a915-450990214859\") " pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.299256 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.299215 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-model-cache\") pod \"test-model-b30a14-kserve-7f5bcc8d4f-79gzw\" (UID: \"49089888-27d7-482d-a915-450990214859\") " pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.300195 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.300132 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49089888-27d7-482d-a915-450990214859-tls-certs\") pod \"test-model-b30a14-kserve-7f5bcc8d4f-79gzw\" (UID: \"49089888-27d7-482d-a915-450990214859\") " pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.305150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.304486 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-dshm\") pod \"test-model-b30a14-kserve-7f5bcc8d4f-79gzw\" (UID: \"49089888-27d7-482d-a915-450990214859\") " pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.348631 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.348589 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brkrw\" (UniqueName: \"kubernetes.io/projected/49089888-27d7-482d-a915-450990214859-kube-api-access-brkrw\") pod \"test-model-b30a14-kserve-7f5bcc8d4f-79gzw\" (UID: \"49089888-27d7-482d-a915-450990214859\") " pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:14.649193 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:14.649057 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:15.045479 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:15.045444 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw"] Nov 03 22:46:15.045902 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:46:15.045871 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49089888_27d7_482d_a915_450990214859.slice/crio-08496b308367305fa4a030f5d5c7d652b3a88d8d980206ba5bdbd3b1efef10b9 WatchSource:0}: Error finding container 08496b308367305fa4a030f5d5c7d652b3a88d8d980206ba5bdbd3b1efef10b9: Status 404 returned error can't find the container with id 08496b308367305fa4a030f5d5c7d652b3a88d8d980206ba5bdbd3b1efef10b9 Nov 03 22:46:15.905654 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:15.905613 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" event={"ID":"49089888-27d7-482d-a915-450990214859","Type":"ContainerStarted","Data":"030da696d2d7801525333182f4e2a74c1ce59a1074769ea396d1ef88cc939e95"} Nov 03 22:46:15.905887 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:15.905664 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" event={"ID":"49089888-27d7-482d-a915-450990214859","Type":"ContainerStarted","Data":"08496b308367305fa4a030f5d5c7d652b3a88d8d980206ba5bdbd3b1efef10b9"} Nov 03 22:46:16.023608 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:16.023534 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" podStartSLOduration=3.023513317 podStartE2EDuration="3.023513317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:16.02169761 +0000 UTC m=+2343.615729585" watchObservedRunningTime="2026-11-03 22:46:16.023513317 +0000 UTC m=+2343.617545305" Nov 03 22:46:16.659083 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:16.654018 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:24.649740 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:24.649663 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:24.662971 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:24.662939 5084 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:24.995676 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:24.995587 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:46:42.023564 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:46:42.023525 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:46:43.011582 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:43.011536 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7"] Nov 03 22:46:43.016163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:43.016136 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:46:43.070163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:43.070128 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7"] Nov 03 22:46:43.104535 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:43.104488 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/429389e2-f479-4c31-9170-bf11ff231d48-maas-api-tls\") pod \"maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7\" (UID: \"429389e2-f479-4c31-9170-bf11ff231d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:46:43.104759 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:43.104563 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/429389e2-f479-4c31-9170-bf11ff231d48-metrics-certs\") pod \"maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7\" (UID: \"429389e2-f479-4c31-9170-bf11ff231d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:46:43.104843 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:43.104789 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbx6w\" (UniqueName: \"kubernetes.io/projected/429389e2-f479-4c31-9170-bf11ff231d48-kube-api-access-fbx6w\") pod \"maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7\" (UID: \"429389e2-f479-4c31-9170-bf11ff231d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:46:43.104985 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:43.104957 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/429389e2-f479-4c31-9170-bf11ff231d48-openshift-service-ca\") pod \"maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7\" (UID: \"429389e2-f479-4c31-9170-bf11ff231d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:46:43.116261 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:43.116218 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-7bcd0b-serving-cert\"" Nov 03 22:46:43.205989 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:43.205940 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbx6w\" (UniqueName: \"kubernetes.io/projected/429389e2-f479-4c31-9170-bf11ff231d48-kube-api-access-fbx6w\") pod \"maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7\" (UID: \"429389e2-f479-4c31-9170-bf11ff231d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:46:43.206247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:43.206055 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/429389e2-f479-4c31-9170-bf11ff231d48-openshift-service-ca\") pod \"maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7\" (UID: \"429389e2-f479-4c31-9170-bf11ff231d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:46:43.206247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:43.206173 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/429389e2-f479-4c31-9170-bf11ff231d48-maas-api-tls\") pod \"maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7\" (UID: \"429389e2-f479-4c31-9170-bf11ff231d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:46:43.206247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:43.206202 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/429389e2-f479-4c31-9170-bf11ff231d48-metrics-certs\") pod \"maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7\" (UID: \"429389e2-f479-4c31-9170-bf11ff231d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:46:43.206442 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:46:43.206339 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-serving-cert: secret "maas-api-e2e-api-b-7bcd0b-serving-cert" not found Nov 03 22:46:43.206442 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:46:43.206427 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429389e2-f479-4c31-9170-bf11ff231d48-maas-api-tls podName:429389e2-f479-4c31-9170-bf11ff231d48 nodeName:}" failed. No retries permitted until 2026-11-03 22:46:43.706404314 +0000 UTC m=+2371.300436281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/429389e2-f479-4c31-9170-bf11ff231d48-maas-api-tls") pod "maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" (UID: "429389e2-f479-4c31-9170-bf11ff231d48") : secret "maas-api-e2e-api-b-7bcd0b-serving-cert" not found Nov 03 22:46:43.206838 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:43.206814 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/429389e2-f479-4c31-9170-bf11ff231d48-openshift-service-ca\") pod \"maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7\" (UID: \"429389e2-f479-4c31-9170-bf11ff231d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:46:43.208769 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:43.208743 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/429389e2-f479-4c31-9170-bf11ff231d48-metrics-certs\") pod \"maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7\" (UID: \"429389e2-f479-4c31-9170-bf11ff231d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:46:43.288480 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:43.288443 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbx6w\" (UniqueName: \"kubernetes.io/projected/429389e2-f479-4c31-9170-bf11ff231d48-kube-api-access-fbx6w\") pod \"maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7\" (UID: \"429389e2-f479-4c31-9170-bf11ff231d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:46:43.715846 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:43.715743 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/429389e2-f479-4c31-9170-bf11ff231d48-maas-api-tls\") pod \"maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7\" (UID: \"429389e2-f479-4c31-9170-bf11ff231d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:46:43.718549 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:43.718520 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/429389e2-f479-4c31-9170-bf11ff231d48-maas-api-tls\") pod \"maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7\" (UID: \"429389e2-f479-4c31-9170-bf11ff231d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:46:43.932572 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:43.932538 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:46:44.315568 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:44.315533 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7"] Nov 03 22:46:44.316521 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:46:44.316495 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429389e2_f479_4c31_9170_bf11ff231d48.slice/crio-0f227412d8b773dd9a7ab3335662df4859fb79ef6c38e2d8f3825226990b6e64 WatchSource:0}: Error finding container 0f227412d8b773dd9a7ab3335662df4859fb79ef6c38e2d8f3825226990b6e64: Status 404 returned error can't find the container with id 0f227412d8b773dd9a7ab3335662df4859fb79ef6c38e2d8f3825226990b6e64 Nov 03 22:46:44.318004 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:44.317986 5084 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:46:45.148170 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:45.148111 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" event={"ID":"429389e2-f479-4c31-9170-bf11ff231d48","Type":"ContainerStarted","Data":"0f227412d8b773dd9a7ab3335662df4859fb79ef6c38e2d8f3825226990b6e64"} Nov 03 22:46:46.241090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:46.240325 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" event={"ID":"429389e2-f479-4c31-9170-bf11ff231d48","Type":"ContainerStarted","Data":"9726ea99f773993eedec23e63909f6fef54878b04ee5ef05bda9f46a8d8bb7e2"} Nov 03 22:46:46.241675 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:46.241597 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:46:46.357857 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:46.357749 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" podStartSLOduration=3.434447589 podStartE2EDuration="4.357730654s" totalImagesPullingTime="923.283065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:46.357414616 +0000 UTC m=+2373.951446589" watchObservedRunningTime="2026-11-03 22:46:46.357730654 +0000 UTC m=+2373.951762631" Nov 03 22:46:53.263463 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:53.263362 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:46:57.445584 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:57.445543 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7"] Nov 03 22:46:57.446144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:57.445819 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" podUID="49b09b05-12bc-4377-9b4a-325eac04f4c0" containerName="istio-proxy" containerID="cri-o://e04289dc44a39530e25dc5ce11ed1cd6cfe19ba13e7e43c0e5670c738e9d59d5" gracePeriod=30 Nov 03 22:46:57.914887 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:57.914831 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz"] Nov 03 22:46:57.922170 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:57.922138 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:46:57.976898 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:57.976862 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-295682-serving-cert\"" Nov 03 22:46:57.979688 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:57.979649 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz"] Nov 03 22:46:58.039143 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:58.039106 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2ea5507-0676-4e91-a1fd-b13b524cd473-maas-api-tls\") pod \"maas-api-e2e-api-b-295682-848544cd6b-2g5cz\" (UID: \"f2ea5507-0676-4e91-a1fd-b13b524cd473\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:46:58.039325 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:58.039286 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2ea5507-0676-4e91-a1fd-b13b524cd473-metrics-certs\") pod \"maas-api-e2e-api-b-295682-848544cd6b-2g5cz\" (UID: \"f2ea5507-0676-4e91-a1fd-b13b524cd473\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:46:58.039325 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:58.039318 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pvt4\" (UniqueName: \"kubernetes.io/projected/f2ea5507-0676-4e91-a1fd-b13b524cd473-kube-api-access-9pvt4\") pod \"maas-api-e2e-api-b-295682-848544cd6b-2g5cz\" (UID: \"f2ea5507-0676-4e91-a1fd-b13b524cd473\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:46:58.039413 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:58.039372 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ea5507-0676-4e91-a1fd-b13b524cd473-openshift-service-ca\") pod \"maas-api-e2e-api-b-295682-848544cd6b-2g5cz\" (UID: \"f2ea5507-0676-4e91-a1fd-b13b524cd473\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:46:58.140888 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:58.140837 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2ea5507-0676-4e91-a1fd-b13b524cd473-metrics-certs\") pod \"maas-api-e2e-api-b-295682-848544cd6b-2g5cz\" (UID: \"f2ea5507-0676-4e91-a1fd-b13b524cd473\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:46:58.141101 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:58.140910 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pvt4\" (UniqueName: \"kubernetes.io/projected/f2ea5507-0676-4e91-a1fd-b13b524cd473-kube-api-access-9pvt4\") pod \"maas-api-e2e-api-b-295682-848544cd6b-2g5cz\" (UID: \"f2ea5507-0676-4e91-a1fd-b13b524cd473\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:46:58.141101 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:58.141015 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ea5507-0676-4e91-a1fd-b13b524cd473-openshift-service-ca\") pod \"maas-api-e2e-api-b-295682-848544cd6b-2g5cz\" (UID: \"f2ea5507-0676-4e91-a1fd-b13b524cd473\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:46:58.141253 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:58.141119 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2ea5507-0676-4e91-a1fd-b13b524cd473-maas-api-tls\") pod \"maas-api-e2e-api-b-295682-848544cd6b-2g5cz\" (UID: \"f2ea5507-0676-4e91-a1fd-b13b524cd473\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:46:58.141315 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:46:58.141266 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-295682-serving-cert: secret "maas-api-e2e-api-b-295682-serving-cert" not found Nov 03 22:46:58.141375 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:46:58.141338 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2ea5507-0676-4e91-a1fd-b13b524cd473-maas-api-tls podName:f2ea5507-0676-4e91-a1fd-b13b524cd473 nodeName:}" failed. No retries permitted until 2026-11-03 22:46:58.641317203 +0000 UTC m=+2386.235349155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2ea5507-0676-4e91-a1fd-b13b524cd473-maas-api-tls") pod "maas-api-e2e-api-b-295682-848544cd6b-2g5cz" (UID: "f2ea5507-0676-4e91-a1fd-b13b524cd473") : secret "maas-api-e2e-api-b-295682-serving-cert" not found Nov 03 22:46:58.141844 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:58.141816 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ea5507-0676-4e91-a1fd-b13b524cd473-openshift-service-ca\") pod \"maas-api-e2e-api-b-295682-848544cd6b-2g5cz\" (UID: \"f2ea5507-0676-4e91-a1fd-b13b524cd473\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:46:58.144124 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:58.144094 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2ea5507-0676-4e91-a1fd-b13b524cd473-metrics-certs\") pod \"maas-api-e2e-api-b-295682-848544cd6b-2g5cz\" (UID: \"f2ea5507-0676-4e91-a1fd-b13b524cd473\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:46:58.201870 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:58.201775 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pvt4\" (UniqueName: \"kubernetes.io/projected/f2ea5507-0676-4e91-a1fd-b13b524cd473-kube-api-access-9pvt4\") pod \"maas-api-e2e-api-b-295682-848544cd6b-2g5cz\" (UID: \"f2ea5507-0676-4e91-a1fd-b13b524cd473\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:46:58.650433 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:58.650390 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2ea5507-0676-4e91-a1fd-b13b524cd473-maas-api-tls\") pod \"maas-api-e2e-api-b-295682-848544cd6b-2g5cz\" (UID: \"f2ea5507-0676-4e91-a1fd-b13b524cd473\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:46:58.653153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:58.653126 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2ea5507-0676-4e91-a1fd-b13b524cd473-maas-api-tls\") pod \"maas-api-e2e-api-b-295682-848544cd6b-2g5cz\" (UID: \"f2ea5507-0676-4e91-a1fd-b13b524cd473\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:46:58.842930 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:58.842883 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:46:59.242322 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:59.242286 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz"] Nov 03 22:46:59.242718 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:46:59.242693 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ea5507_0676_4e91_a1fd_b13b524cd473.slice/crio-a8c6905760c5978a51a921f0adf89c3f2d1d6e2174e35978da6fe042d59a2fc2 WatchSource:0}: Error finding container a8c6905760c5978a51a921f0adf89c3f2d1d6e2174e35978da6fe042d59a2fc2: Status 404 returned error can't find the container with id a8c6905760c5978a51a921f0adf89c3f2d1d6e2174e35978da6fe042d59a2fc2 Nov 03 22:46:59.408455 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:59.408414 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" event={"ID":"f2ea5507-0676-4e91-a1fd-b13b524cd473","Type":"ContainerStarted","Data":"a8c6905760c5978a51a921f0adf89c3f2d1d6e2174e35978da6fe042d59a2fc2"} Nov 03 22:46:59.626903 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:59.626867 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw"] Nov 03 22:46:59.627153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:59.627118 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" podUID="c05a9c50-a87e-4fa7-b3d7-c06557223767" containerName="istio-proxy" containerID="cri-o://6b6af7f7a43bfc14fe1e79676dd7959217a4de781bb17c20fbe8d56f44f5498e" gracePeriod=30 Nov 03 22:46:59.739788 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:59.739738 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj"] Nov 03 22:46:59.740283 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:46:59.740015 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" podUID="892a000a-c16a-4904-8d6d-57952348afc9" containerName="istio-proxy" containerID="cri-o://1383950ff17acdec04cfb6f1f9e4cffb3c04762f4e5f64f657b253764fc9e172" gracePeriod=30 Nov 03 22:47:00.257784 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:00.257740 5084 patch_prober.go:29] interesting pod/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:47:00.257975 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:00.257821 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" podUID="c05a9c50-a87e-4fa7-b3d7-c06557223767" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:47:00.418204 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:00.418159 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" event={"ID":"f2ea5507-0676-4e91-a1fd-b13b524cd473","Type":"ContainerStarted","Data":"e9b3fcc0ec2f85f254c12e9dc5cb8676bf3cdedcd12b3e8997137e9c3f2a389c"} Nov 03 22:47:00.418419 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:00.418238 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:47:00.537659 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:00.537601 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" podStartSLOduration=2.596493541 podStartE2EDuration="3.537584083s" totalImagesPullingTime="941.090542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:00.534197466 +0000 UTC m=+2388.128229439" watchObservedRunningTime="2026-11-03 22:47:00.537584083 +0000 UTC m=+2388.131616057" Nov 03 22:47:02.709133 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:02.709051 5084 patch_prober.go:29] interesting pod/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:47:02.709683 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:02.709178 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" podUID="892a000a-c16a-4904-8d6d-57952348afc9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:47:02.890416 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:02.890369 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:47:03.040040 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.039997 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-data\") pod \"49b09b05-12bc-4377-9b4a-325eac04f4c0\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " Nov 03 22:47:03.040274 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.040057 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-credential-socket\") pod \"49b09b05-12bc-4377-9b4a-325eac04f4c0\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " Nov 03 22:47:03.040274 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.040139 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49b09b05-12bc-4377-9b4a-325eac04f4c0-kube-api-access-pcskd\" (UniqueName: \"kubernetes.io/projected/49b09b05-12bc-4377-9b4a-325eac04f4c0-kube-api-access-pcskd\") pod \"49b09b05-12bc-4377-9b4a-325eac04f4c0\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " Nov 03 22:47:03.040274 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.040188 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-workload-certs\") pod \"49b09b05-12bc-4377-9b4a-325eac04f4c0\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " Nov 03 22:47:03.040274 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.040269 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49b09b05-12bc-4377-9b4a-325eac04f4c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b09b05-12bc-4377-9b4a-325eac04f4c0-istiod-ca-cert\") pod \"49b09b05-12bc-4377-9b4a-325eac04f4c0\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " Nov 03 22:47:03.040489 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.040396 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-workload-socket\") pod \"49b09b05-12bc-4377-9b4a-325eac04f4c0\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " Nov 03 22:47:03.040489 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.040396 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-data" pod "49b09b05-12bc-4377-9b4a-325eac04f4c0" (UID: "49b09b05-12bc-4377-9b4a-325eac04f4c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:03.040489 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.040416 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-credential-socket" pod "49b09b05-12bc-4377-9b4a-325eac04f4c0" (UID: "49b09b05-12bc-4377-9b4a-325eac04f4c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:03.040628 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.040493 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-envoy\") pod \"49b09b05-12bc-4377-9b4a-325eac04f4c0\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " Nov 03 22:47:03.040628 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.040570 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-token\" (UniqueName: \"kubernetes.io/projected/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-token\") pod \"49b09b05-12bc-4377-9b4a-325eac04f4c0\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " Nov 03 22:47:03.040746 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.040636 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-podinfo\") pod \"49b09b05-12bc-4377-9b4a-325eac04f4c0\" (UID: \"49b09b05-12bc-4377-9b4a-325eac04f4c0\") " Nov 03 22:47:03.040844 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.040633 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-workload-certs" pod "49b09b05-12bc-4377-9b4a-325eac04f4c0" (UID: "49b09b05-12bc-4377-9b4a-325eac04f4c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:03.040844 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.040679 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-workload-socket" pod "49b09b05-12bc-4377-9b4a-325eac04f4c0" (UID: "49b09b05-12bc-4377-9b4a-325eac04f4c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:03.040844 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.040698 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b09b05-12bc-4377-9b4a-325eac04f4c0-istiod-ca-cert" pod "49b09b05-12bc-4377-9b4a-325eac04f4c0" (UID: "49b09b05-12bc-4377-9b4a-325eac04f4c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:03.041459 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.041431 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:03.041597 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.041461 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:03.041597 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.041478 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:03.041597 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.041492 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49b09b05-12bc-4377-9b4a-325eac04f4c0-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:03.041597 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.041505 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:03.042894 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.042862 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-envoy" pod "49b09b05-12bc-4377-9b4a-325eac04f4c0" (UID: "49b09b05-12bc-4377-9b4a-325eac04f4c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:03.043021 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.042974 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b09b05-12bc-4377-9b4a-325eac04f4c0-kube-api-access-pcskd" pod "49b09b05-12bc-4377-9b4a-325eac04f4c0" (UID: "49b09b05-12bc-4377-9b4a-325eac04f4c0"). InnerVolumeSpecName "kube-api-access-pcskd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:03.043279 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.043252 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-podinfo" pod "49b09b05-12bc-4377-9b4a-325eac04f4c0" (UID: "49b09b05-12bc-4377-9b4a-325eac04f4c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:03.043445 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.043428 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-token" pod "49b09b05-12bc-4377-9b4a-325eac04f4c0" (UID: "49b09b05-12bc-4377-9b4a-325eac04f4c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:03.142495 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.142454 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:03.142495 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.142488 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:03.142495 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.142498 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49b09b05-12bc-4377-9b4a-325eac04f4c0-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:03.142887 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.142508 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcskd\" (UniqueName: \"kubernetes.io/projected/49b09b05-12bc-4377-9b4a-325eac04f4c0-kube-api-access-pcskd\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:03.445834 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.445725 5084 generic.go:356] "Generic (PLEG): container finished" podID="49b09b05-12bc-4377-9b4a-325eac04f4c0" containerID="e04289dc44a39530e25dc5ce11ed1cd6cfe19ba13e7e43c0e5670c738e9d59d5" exitCode=0 Nov 03 22:47:03.445834 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.445809 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" Nov 03 22:47:03.446111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.445841 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" event={"ID":"49b09b05-12bc-4377-9b4a-325eac04f4c0","Type":"ContainerDied","Data":"e04289dc44a39530e25dc5ce11ed1cd6cfe19ba13e7e43c0e5670c738e9d59d5"} Nov 03 22:47:03.446111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.445911 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7" event={"ID":"49b09b05-12bc-4377-9b4a-325eac04f4c0","Type":"ContainerDied","Data":"64a3d305755862490f761a9dc65439c774c2717a4a16a732cc1d7df3e969f26f"} Nov 03 22:47:03.446111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.445951 5084 scope.go:122] "RemoveContainer" containerID="e04289dc44a39530e25dc5ce11ed1cd6cfe19ba13e7e43c0e5670c738e9d59d5" Nov 03 22:47:03.460546 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.460512 5084 scope.go:122] "RemoveContainer" containerID="e04289dc44a39530e25dc5ce11ed1cd6cfe19ba13e7e43c0e5670c738e9d59d5" Nov 03 22:47:03.461138 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:47:03.461007 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04289dc44a39530e25dc5ce11ed1cd6cfe19ba13e7e43c0e5670c738e9d59d5\": container with ID starting with e04289dc44a39530e25dc5ce11ed1cd6cfe19ba13e7e43c0e5670c738e9d59d5 not found: ID does not exist" containerID="e04289dc44a39530e25dc5ce11ed1cd6cfe19ba13e7e43c0e5670c738e9d59d5" Nov 03 22:47:03.461138 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.461057 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04289dc44a39530e25dc5ce11ed1cd6cfe19ba13e7e43c0e5670c738e9d59d5"} err="failed to get container status \"e04289dc44a39530e25dc5ce11ed1cd6cfe19ba13e7e43c0e5670c738e9d59d5\": rpc error: code = NotFound desc = could not find container \"e04289dc44a39530e25dc5ce11ed1cd6cfe19ba13e7e43c0e5670c738e9d59d5\": container with ID starting with e04289dc44a39530e25dc5ce11ed1cd6cfe19ba13e7e43c0e5670c738e9d59d5 not found: ID does not exist" Nov 03 22:47:03.643311 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.643266 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7"] Nov 03 22:47:03.647892 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:03.647854 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-zqwd7"] Nov 03 22:47:04.671095 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:04.670024 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49b09b05-12bc-4377-9b4a-325eac04f4c0" path="/var/lib/kubelet/pods/49b09b05-12bc-4377-9b4a-325eac04f4c0/volumes" Nov 03 22:47:04.958730 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:04.958689 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:47:05.045143 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.045106 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:47:05.089880 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.089832 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-token\" (UniqueName: \"kubernetes.io/projected/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-token\") pod \"c05a9c50-a87e-4fa7-b3d7-c06557223767\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " Nov 03 22:47:05.090112 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.089932 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-podinfo\") pod \"c05a9c50-a87e-4fa7-b3d7-c06557223767\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " Nov 03 22:47:05.090112 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.090059 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-envoy\") pod \"c05a9c50-a87e-4fa7-b3d7-c06557223767\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " Nov 03 22:47:05.090245 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.090164 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c05a9c50-a87e-4fa7-b3d7-c06557223767-kube-api-access-s7vdb\" (UniqueName: \"kubernetes.io/projected/c05a9c50-a87e-4fa7-b3d7-c06557223767-kube-api-access-s7vdb\") pod \"c05a9c50-a87e-4fa7-b3d7-c06557223767\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " Nov 03 22:47:05.090245 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.090203 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-data\") pod \"c05a9c50-a87e-4fa7-b3d7-c06557223767\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " Nov 03 22:47:05.090245 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.090241 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-workload-certs\") pod \"c05a9c50-a87e-4fa7-b3d7-c06557223767\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " Nov 03 22:47:05.090422 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.090270 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c05a9c50-a87e-4fa7-b3d7-c06557223767-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c05a9c50-a87e-4fa7-b3d7-c06557223767-istiod-ca-cert\") pod \"c05a9c50-a87e-4fa7-b3d7-c06557223767\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " Nov 03 22:47:05.090422 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.090340 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-workload-socket\") pod \"c05a9c50-a87e-4fa7-b3d7-c06557223767\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " Nov 03 22:47:05.090422 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.090376 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-credential-socket\") pod \"c05a9c50-a87e-4fa7-b3d7-c06557223767\" (UID: \"c05a9c50-a87e-4fa7-b3d7-c06557223767\") " Nov 03 22:47:05.090582 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.090520 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-workload-certs" pod "c05a9c50-a87e-4fa7-b3d7-c06557223767" (UID: "c05a9c50-a87e-4fa7-b3d7-c06557223767"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:05.090582 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.090528 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-data" pod "c05a9c50-a87e-4fa7-b3d7-c06557223767" (UID: "c05a9c50-a87e-4fa7-b3d7-c06557223767"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:05.090772 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.090734 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-workload-socket" pod "c05a9c50-a87e-4fa7-b3d7-c06557223767" (UID: "c05a9c50-a87e-4fa7-b3d7-c06557223767"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:05.090842 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.090823 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05a9c50-a87e-4fa7-b3d7-c06557223767-istiod-ca-cert" pod "c05a9c50-a87e-4fa7-b3d7-c06557223767" (UID: "c05a9c50-a87e-4fa7-b3d7-c06557223767"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:05.091003 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.090976 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-credential-socket" pod "c05a9c50-a87e-4fa7-b3d7-c06557223767" (UID: "c05a9c50-a87e-4fa7-b3d7-c06557223767"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:05.091245 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.091222 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:05.091333 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.091251 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:05.091333 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.091267 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c05a9c50-a87e-4fa7-b3d7-c06557223767-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:05.091333 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.091284 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:05.091333 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.091299 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:05.092594 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.092566 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-podinfo" pod "c05a9c50-a87e-4fa7-b3d7-c06557223767" (UID: "c05a9c50-a87e-4fa7-b3d7-c06557223767"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:05.092751 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.092733 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-envoy" pod "c05a9c50-a87e-4fa7-b3d7-c06557223767" (UID: "c05a9c50-a87e-4fa7-b3d7-c06557223767"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:05.092797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.092747 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05a9c50-a87e-4fa7-b3d7-c06557223767-kube-api-access-s7vdb" pod "c05a9c50-a87e-4fa7-b3d7-c06557223767" (UID: "c05a9c50-a87e-4fa7-b3d7-c06557223767"). InnerVolumeSpecName "kube-api-access-s7vdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:05.092797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.092762 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-token" pod "c05a9c50-a87e-4fa7-b3d7-c06557223767" (UID: "c05a9c50-a87e-4fa7-b3d7-c06557223767"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:05.192675 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.192577 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-workload-certs\") pod \"892a000a-c16a-4904-8d6d-57952348afc9\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " Nov 03 22:47:05.192675 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.192648 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892a000a-c16a-4904-8d6d-57952348afc9-istio-token\" (UniqueName: \"kubernetes.io/projected/892a000a-c16a-4904-8d6d-57952348afc9-istio-token\") pod \"892a000a-c16a-4904-8d6d-57952348afc9\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " Nov 03 22:47:05.192935 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.192702 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/892a000a-c16a-4904-8d6d-57952348afc9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892a000a-c16a-4904-8d6d-57952348afc9-istiod-ca-cert\") pod \"892a000a-c16a-4904-8d6d-57952348afc9\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " Nov 03 22:47:05.192935 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.192761 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-credential-socket\") pod \"892a000a-c16a-4904-8d6d-57952348afc9\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " Nov 03 22:47:05.192935 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.192893 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-workload-socket\") pod \"892a000a-c16a-4904-8d6d-57952348afc9\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " Nov 03 22:47:05.193131 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.192931 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892a000a-c16a-4904-8d6d-57952348afc9-kube-api-access-4znxj\" (UniqueName: \"kubernetes.io/projected/892a000a-c16a-4904-8d6d-57952348afc9-kube-api-access-4znxj\") pod \"892a000a-c16a-4904-8d6d-57952348afc9\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " Nov 03 22:47:05.193131 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.192934 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-workload-certs" pod "892a000a-c16a-4904-8d6d-57952348afc9" (UID: "892a000a-c16a-4904-8d6d-57952348afc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:05.193131 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.193001 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/892a000a-c16a-4904-8d6d-57952348afc9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892a000a-c16a-4904-8d6d-57952348afc9-istio-podinfo\") pod \"892a000a-c16a-4904-8d6d-57952348afc9\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " Nov 03 22:47:05.193313 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.193118 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-credential-socket" pod "892a000a-c16a-4904-8d6d-57952348afc9" (UID: "892a000a-c16a-4904-8d6d-57952348afc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:05.193313 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.193143 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-istio-envoy\") pod \"892a000a-c16a-4904-8d6d-57952348afc9\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " Nov 03 22:47:05.193313 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.193166 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892a000a-c16a-4904-8d6d-57952348afc9-istiod-ca-cert" pod "892a000a-c16a-4904-8d6d-57952348afc9" (UID: "892a000a-c16a-4904-8d6d-57952348afc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:05.193313 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.193220 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-istio-data\") pod \"892a000a-c16a-4904-8d6d-57952348afc9\" (UID: \"892a000a-c16a-4904-8d6d-57952348afc9\") " Nov 03 22:47:05.193313 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.193273 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-workload-socket" pod "892a000a-c16a-4904-8d6d-57952348afc9" (UID: "892a000a-c16a-4904-8d6d-57952348afc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:05.193601 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.193582 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-istio-data" pod "892a000a-c16a-4904-8d6d-57952348afc9" (UID: "892a000a-c16a-4904-8d6d-57952348afc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:05.193894 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.193870 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:05.193894 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.193895 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7vdb\" (UniqueName: \"kubernetes.io/projected/c05a9c50-a87e-4fa7-b3d7-c06557223767-kube-api-access-s7vdb\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:05.194150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.193906 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:05.194150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.193915 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892a000a-c16a-4904-8d6d-57952348afc9-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:05.194150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.193924 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:05.194150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.193933 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:05.194150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.193941 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:05.194150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.193950 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:05.194150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.193964 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c05a9c50-a87e-4fa7-b3d7-c06557223767-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:05.195331 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.195307 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/892a000a-c16a-4904-8d6d-57952348afc9-istio-podinfo" pod "892a000a-c16a-4904-8d6d-57952348afc9" (UID: "892a000a-c16a-4904-8d6d-57952348afc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:05.195489 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.195468 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-istio-envoy" pod "892a000a-c16a-4904-8d6d-57952348afc9" (UID: "892a000a-c16a-4904-8d6d-57952348afc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:05.195613 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.195592 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892a000a-c16a-4904-8d6d-57952348afc9-istio-token" pod "892a000a-c16a-4904-8d6d-57952348afc9" (UID: "892a000a-c16a-4904-8d6d-57952348afc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:05.195670 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.195628 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892a000a-c16a-4904-8d6d-57952348afc9-kube-api-access-4znxj" pod "892a000a-c16a-4904-8d6d-57952348afc9" (UID: "892a000a-c16a-4904-8d6d-57952348afc9"). InnerVolumeSpecName "kube-api-access-4znxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:05.295691 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.295581 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892a000a-c16a-4904-8d6d-57952348afc9-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:05.295691 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.295623 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4znxj\" (UniqueName: \"kubernetes.io/projected/892a000a-c16a-4904-8d6d-57952348afc9-kube-api-access-4znxj\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:05.295691 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.295640 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892a000a-c16a-4904-8d6d-57952348afc9-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:05.295691 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.295653 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892a000a-c16a-4904-8d6d-57952348afc9-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:05.466449 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.466341 5084 generic.go:356] "Generic (PLEG): container finished" podID="892a000a-c16a-4904-8d6d-57952348afc9" containerID="1383950ff17acdec04cfb6f1f9e4cffb3c04762f4e5f64f657b253764fc9e172" exitCode=0 Nov 03 22:47:05.466449 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.466404 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" Nov 03 22:47:05.466449 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.466431 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" event={"ID":"892a000a-c16a-4904-8d6d-57952348afc9","Type":"ContainerDied","Data":"1383950ff17acdec04cfb6f1f9e4cffb3c04762f4e5f64f657b253764fc9e172"} Nov 03 22:47:05.466731 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.466494 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj" event={"ID":"892a000a-c16a-4904-8d6d-57952348afc9","Type":"ContainerDied","Data":"042c39debde0478e4cde78e6938ebbf37f9d3b6b1a98548f6a7fddb971459146"} Nov 03 22:47:05.466731 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.466520 5084 scope.go:122] "RemoveContainer" containerID="1383950ff17acdec04cfb6f1f9e4cffb3c04762f4e5f64f657b253764fc9e172" Nov 03 22:47:05.468670 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.468641 5084 generic.go:356] "Generic (PLEG): container finished" podID="c05a9c50-a87e-4fa7-b3d7-c06557223767" containerID="6b6af7f7a43bfc14fe1e79676dd7959217a4de781bb17c20fbe8d56f44f5498e" exitCode=0 Nov 03 22:47:05.468820 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.468706 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" Nov 03 22:47:05.468820 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.468720 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" event={"ID":"c05a9c50-a87e-4fa7-b3d7-c06557223767","Type":"ContainerDied","Data":"6b6af7f7a43bfc14fe1e79676dd7959217a4de781bb17c20fbe8d56f44f5498e"} Nov 03 22:47:05.468820 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.468769 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw" event={"ID":"c05a9c50-a87e-4fa7-b3d7-c06557223767","Type":"ContainerDied","Data":"c768ab936a1064a54e901ac19a34a961bb6b8827a654a8b321f1f7fe523963ab"} Nov 03 22:47:05.479216 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.479190 5084 scope.go:122] "RemoveContainer" containerID="d0bb7ead7a80a2c1536ed2bb792c08d8684a79f21074f865b9febe9ff55c8246" Nov 03 22:47:05.491648 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.491618 5084 scope.go:122] "RemoveContainer" containerID="1383950ff17acdec04cfb6f1f9e4cffb3c04762f4e5f64f657b253764fc9e172" Nov 03 22:47:05.492030 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:47:05.492008 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1383950ff17acdec04cfb6f1f9e4cffb3c04762f4e5f64f657b253764fc9e172\": container with ID starting with 1383950ff17acdec04cfb6f1f9e4cffb3c04762f4e5f64f657b253764fc9e172 not found: ID does not exist" containerID="1383950ff17acdec04cfb6f1f9e4cffb3c04762f4e5f64f657b253764fc9e172" Nov 03 22:47:05.492123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.492041 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1383950ff17acdec04cfb6f1f9e4cffb3c04762f4e5f64f657b253764fc9e172"} err="failed to get container status \"1383950ff17acdec04cfb6f1f9e4cffb3c04762f4e5f64f657b253764fc9e172\": rpc error: code = NotFound desc = could not find container \"1383950ff17acdec04cfb6f1f9e4cffb3c04762f4e5f64f657b253764fc9e172\": container with ID starting with 1383950ff17acdec04cfb6f1f9e4cffb3c04762f4e5f64f657b253764fc9e172 not found: ID does not exist" Nov 03 22:47:05.492123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.492077 5084 scope.go:122] "RemoveContainer" containerID="d0bb7ead7a80a2c1536ed2bb792c08d8684a79f21074f865b9febe9ff55c8246" Nov 03 22:47:05.492378 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:47:05.492358 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0bb7ead7a80a2c1536ed2bb792c08d8684a79f21074f865b9febe9ff55c8246\": container with ID starting with d0bb7ead7a80a2c1536ed2bb792c08d8684a79f21074f865b9febe9ff55c8246 not found: ID does not exist" containerID="d0bb7ead7a80a2c1536ed2bb792c08d8684a79f21074f865b9febe9ff55c8246" Nov 03 22:47:05.492427 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.492388 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0bb7ead7a80a2c1536ed2bb792c08d8684a79f21074f865b9febe9ff55c8246"} err="failed to get container status \"d0bb7ead7a80a2c1536ed2bb792c08d8684a79f21074f865b9febe9ff55c8246\": rpc error: code = NotFound desc = could not find container \"d0bb7ead7a80a2c1536ed2bb792c08d8684a79f21074f865b9febe9ff55c8246\": container with ID starting with d0bb7ead7a80a2c1536ed2bb792c08d8684a79f21074f865b9febe9ff55c8246 not found: ID does not exist" Nov 03 22:47:05.492427 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.492406 5084 scope.go:122] "RemoveContainer" containerID="6b6af7f7a43bfc14fe1e79676dd7959217a4de781bb17c20fbe8d56f44f5498e" Nov 03 22:47:05.503577 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.503544 5084 scope.go:122] "RemoveContainer" containerID="6b6af7f7a43bfc14fe1e79676dd7959217a4de781bb17c20fbe8d56f44f5498e" Nov 03 22:47:05.503962 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:47:05.503937 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6af7f7a43bfc14fe1e79676dd7959217a4de781bb17c20fbe8d56f44f5498e\": container with ID starting with 6b6af7f7a43bfc14fe1e79676dd7959217a4de781bb17c20fbe8d56f44f5498e not found: ID does not exist" containerID="6b6af7f7a43bfc14fe1e79676dd7959217a4de781bb17c20fbe8d56f44f5498e" Nov 03 22:47:05.504054 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.503972 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6af7f7a43bfc14fe1e79676dd7959217a4de781bb17c20fbe8d56f44f5498e"} err="failed to get container status \"6b6af7f7a43bfc14fe1e79676dd7959217a4de781bb17c20fbe8d56f44f5498e\": rpc error: code = NotFound desc = could not find container \"6b6af7f7a43bfc14fe1e79676dd7959217a4de781bb17c20fbe8d56f44f5498e\": container with ID starting with 6b6af7f7a43bfc14fe1e79676dd7959217a4de781bb17c20fbe8d56f44f5498e not found: ID does not exist" Nov 03 22:47:05.636243 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.636208 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj"] Nov 03 22:47:05.688113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.688054 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-41bb2b-openshift-default-86f696c85-vlbfj"] Nov 03 22:47:05.753493 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.753385 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw"] Nov 03 22:47:05.760052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:05.760011 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-d79nw"] Nov 03 22:47:06.432668 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:06.432638 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:47:06.630869 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:06.630832 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="892a000a-c16a-4904-8d6d-57952348afc9" path="/var/lib/kubelet/pods/892a000a-c16a-4904-8d6d-57952348afc9/volumes" Nov 03 22:47:06.631469 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:06.631452 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c05a9c50-a87e-4fa7-b3d7-c06557223767" path="/var/lib/kubelet/pods/c05a9c50-a87e-4fa7-b3d7-c06557223767/volumes" Nov 03 22:47:12.028180 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:47:12.028134 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:47:13.095521 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:13.095491 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:47:13.096097 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:13.095600 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:47:22.947765 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:22.947725 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp"] Nov 03 22:47:22.948660 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:22.948633 5084 cpu_manager.go:414] "containerMap: removing container" podUID="49b09b05-12bc-4377-9b4a-325eac04f4c0" containerName="istio-proxy" Nov 03 22:47:22.948779 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:22.948661 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b09b05-12bc-4377-9b4a-325eac04f4c0" containerName="istio-proxy" Nov 03 22:47:22.948779 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:22.948682 5084 cpu_manager.go:414] "containerMap: removing container" podUID="892a000a-c16a-4904-8d6d-57952348afc9" containerName="istio-proxy" Nov 03 22:47:22.948779 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:22.948690 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892a000a-c16a-4904-8d6d-57952348afc9" containerName="istio-proxy" Nov 03 22:47:22.948779 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:22.948746 5084 cpu_manager.go:414] "containerMap: removing container" podUID="c05a9c50-a87e-4fa7-b3d7-c06557223767" containerName="istio-proxy" Nov 03 22:47:22.948779 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:22.948755 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c05a9c50-a87e-4fa7-b3d7-c06557223767" containerName="istio-proxy" Nov 03 22:47:22.949020 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:22.948931 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="c05a9c50-a87e-4fa7-b3d7-c06557223767" containerName="istio-proxy" Nov 03 22:47:22.949020 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:22.948950 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="892a000a-c16a-4904-8d6d-57952348afc9" containerName="istio-proxy" Nov 03 22:47:22.949020 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:22.948964 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b09b05-12bc-4377-9b4a-325eac04f4c0" containerName="istio-proxy" Nov 03 22:47:22.949020 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:22.948978 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="892a000a-c16a-4904-8d6d-57952348afc9" containerName="istio-proxy" Nov 03 22:47:22.954308 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:22.954276 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:47:23.009001 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:23.008955 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp"] Nov 03 22:47:23.045896 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:23.045839 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx2gh\" (UniqueName: \"kubernetes.io/projected/912af5f3-b029-4368-b74b-693937c9431d-kube-api-access-tx2gh\") pod \"maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp\" (UID: \"912af5f3-b029-4368-b74b-693937c9431d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:47:23.046101 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:23.046041 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/912af5f3-b029-4368-b74b-693937c9431d-metrics-certs\") pod \"maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp\" (UID: \"912af5f3-b029-4368-b74b-693937c9431d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:47:23.046185 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:23.046166 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/912af5f3-b029-4368-b74b-693937c9431d-openshift-service-ca\") pod \"maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp\" (UID: \"912af5f3-b029-4368-b74b-693937c9431d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:47:23.046381 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:23.046350 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/912af5f3-b029-4368-b74b-693937c9431d-maas-api-tls\") pod \"maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp\" (UID: \"912af5f3-b029-4368-b74b-693937c9431d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:47:23.049762 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:23.049722 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-472b46-serving-cert\"" Nov 03 22:47:23.147544 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:23.147491 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/912af5f3-b029-4368-b74b-693937c9431d-openshift-service-ca\") pod \"maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp\" (UID: \"912af5f3-b029-4368-b74b-693937c9431d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:47:23.147756 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:23.147629 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/912af5f3-b029-4368-b74b-693937c9431d-maas-api-tls\") pod \"maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp\" (UID: \"912af5f3-b029-4368-b74b-693937c9431d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:47:23.147756 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:23.147721 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx2gh\" (UniqueName: \"kubernetes.io/projected/912af5f3-b029-4368-b74b-693937c9431d-kube-api-access-tx2gh\") pod \"maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp\" (UID: \"912af5f3-b029-4368-b74b-693937c9431d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:47:23.147864 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:47:23.147757 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-serving-cert: secret "maas-api-e2e-api-b-472b46-serving-cert" not found Nov 03 22:47:23.147864 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:47:23.147833 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/912af5f3-b029-4368-b74b-693937c9431d-maas-api-tls podName:912af5f3-b029-4368-b74b-693937c9431d nodeName:}" failed. No retries permitted until 2026-11-03 22:47:23.64781673 +0000 UTC m=+2411.241848684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/912af5f3-b029-4368-b74b-693937c9431d-maas-api-tls") pod "maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" (UID: "912af5f3-b029-4368-b74b-693937c9431d") : secret "maas-api-e2e-api-b-472b46-serving-cert" not found Nov 03 22:47:23.147983 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:23.147866 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/912af5f3-b029-4368-b74b-693937c9431d-metrics-certs\") pod \"maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp\" (UID: \"912af5f3-b029-4368-b74b-693937c9431d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:47:23.148349 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:23.148314 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/912af5f3-b029-4368-b74b-693937c9431d-openshift-service-ca\") pod \"maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp\" (UID: \"912af5f3-b029-4368-b74b-693937c9431d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:47:23.150440 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:23.150414 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/912af5f3-b029-4368-b74b-693937c9431d-metrics-certs\") pod \"maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp\" (UID: \"912af5f3-b029-4368-b74b-693937c9431d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:47:23.233681 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:23.233542 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx2gh\" (UniqueName: \"kubernetes.io/projected/912af5f3-b029-4368-b74b-693937c9431d-kube-api-access-tx2gh\") pod \"maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp\" (UID: \"912af5f3-b029-4368-b74b-693937c9431d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:47:23.654421 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:23.654373 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/912af5f3-b029-4368-b74b-693937c9431d-maas-api-tls\") pod \"maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp\" (UID: \"912af5f3-b029-4368-b74b-693937c9431d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:47:23.657046 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:23.657013 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/912af5f3-b029-4368-b74b-693937c9431d-maas-api-tls\") pod \"maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp\" (UID: \"912af5f3-b029-4368-b74b-693937c9431d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:47:23.870402 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:23.870358 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:47:24.258896 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:24.258861 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp"] Nov 03 22:47:24.260054 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:47:24.260023 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912af5f3_b029_4368_b74b_693937c9431d.slice/crio-e56d3d539eca0201b191c2b93cd11c7a0c010eb4172cda9004ff159772be8289 WatchSource:0}: Error finding container e56d3d539eca0201b191c2b93cd11c7a0c010eb4172cda9004ff159772be8289: Status 404 returned error can't find the container with id e56d3d539eca0201b191c2b93cd11c7a0c010eb4172cda9004ff159772be8289 Nov 03 22:47:24.637147 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:24.637107 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" event={"ID":"912af5f3-b029-4368-b74b-693937c9431d","Type":"ContainerStarted","Data":"e56d3d539eca0201b191c2b93cd11c7a0c010eb4172cda9004ff159772be8289"} Nov 03 22:47:25.647096 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:25.647008 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" event={"ID":"912af5f3-b029-4368-b74b-693937c9431d","Type":"ContainerStarted","Data":"4fe9b37e4c339022f8d2c9f323878cd25f6f6ecfcccdff749318d48ebe12d92e"} Nov 03 22:47:25.647641 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:25.647147 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:47:31.661291 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:31.661256 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:47:31.781218 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:31.781157 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" podStartSLOduration=8.881835006 podStartE2EDuration="9.7811403s" totalImagesPullingTime="899.305294ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:25.76813075 +0000 UTC m=+2413.362162725" watchObservedRunningTime="2026-11-03 22:47:31.7811403 +0000 UTC m=+2419.375172274" Nov 03 22:47:41.378245 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:41.378200 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm"] Nov 03 22:47:41.379098 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:41.379043 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" podUID="bb5188a0-6554-4eb7-8567-0613709d03d9" containerName="istio-proxy" containerID="cri-o://fcab45c5e325a7cb1fe394921068dca1d24302e10a213fad173912e8b1a58071" gracePeriod=30 Nov 03 22:47:41.401324 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:41.401283 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7"] Nov 03 22:47:41.401547 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:41.401500 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" podUID="135fb3ee-0298-424f-b188-611bdfbec980" containerName="istio-proxy" containerID="cri-o://c2dcd4eedb3bcd36b3f68503ec93e2582e62530a7bed4928e11440a3317282d4" gracePeriod=30 Nov 03 22:47:42.025010 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:47:42.024969 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:47:43.329307 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:43.329269 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9"] Nov 03 22:47:43.330000 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:43.329489 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" podUID="531b6497-408c-4315-806e-0504eead114a" containerName="istio-proxy" containerID="cri-o://43b1c2e494111edaf6ad2141c39562f40b92641dd0c17ebf3eee031ab3e6d162" gracePeriod=30 Nov 03 22:47:43.333196 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:43.333165 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r"] Nov 03 22:47:43.333395 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:43.333368 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" podUID="f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" containerName="istio-proxy" containerID="cri-o://e6fa413bffd0164d8e9dda994e28d080418a7b3b598c4a051b8424c7449ad819" gracePeriod=30 Nov 03 22:47:46.327577 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.327527 5084 patch_prober.go:29] interesting pod/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:47:46.328017 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.327604 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" podUID="531b6497-408c-4315-806e-0504eead114a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:47:46.717898 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.717864 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:47:46.750109 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.750058 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:47:46.810308 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.810255 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-envoy\") pod \"bb5188a0-6554-4eb7-8567-0613709d03d9\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " Nov 03 22:47:46.810532 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.810350 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-workload-certs\") pod \"135fb3ee-0298-424f-b188-611bdfbec980\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " Nov 03 22:47:46.810532 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.810386 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5188a0-6554-4eb7-8567-0613709d03d9-kube-api-access-6w7xz\" (UniqueName: \"kubernetes.io/projected/bb5188a0-6554-4eb7-8567-0613709d03d9-kube-api-access-6w7xz\") pod \"bb5188a0-6554-4eb7-8567-0613709d03d9\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " Nov 03 22:47:46.810532 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.810435 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135fb3ee-0298-424f-b188-611bdfbec980-kube-api-access-xdg5v\" (UniqueName: \"kubernetes.io/projected/135fb3ee-0298-424f-b188-611bdfbec980-kube-api-access-xdg5v\") pod \"135fb3ee-0298-424f-b188-611bdfbec980\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " Nov 03 22:47:46.810532 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.810470 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-podinfo\") pod \"bb5188a0-6554-4eb7-8567-0613709d03d9\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " Nov 03 22:47:46.810765 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.810578 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135fb3ee-0298-424f-b188-611bdfbec980-istio-token\" (UniqueName: \"kubernetes.io/projected/135fb3ee-0298-424f-b188-611bdfbec980-istio-token\") pod \"135fb3ee-0298-424f-b188-611bdfbec980\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " Nov 03 22:47:46.810765 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.810607 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5188a0-6554-4eb7-8567-0613709d03d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5188a0-6554-4eb7-8567-0613709d03d9-istiod-ca-cert\") pod \"bb5188a0-6554-4eb7-8567-0613709d03d9\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " Nov 03 22:47:46.810765 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.810638 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-workload-certs\") pod \"bb5188a0-6554-4eb7-8567-0613709d03d9\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " Nov 03 22:47:46.810765 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.810670 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-workload-socket\") pod \"bb5188a0-6554-4eb7-8567-0613709d03d9\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " Nov 03 22:47:46.810765 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.810712 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-istio-envoy\") pod \"135fb3ee-0298-424f-b188-611bdfbec980\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " Nov 03 22:47:46.810765 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.810744 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-credential-socket\") pod \"bb5188a0-6554-4eb7-8567-0613709d03d9\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " Nov 03 22:47:46.811123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.810811 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/135fb3ee-0298-424f-b188-611bdfbec980-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135fb3ee-0298-424f-b188-611bdfbec980-istio-podinfo\") pod \"135fb3ee-0298-424f-b188-611bdfbec980\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " Nov 03 22:47:46.811123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.810840 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-istio-data\") pod \"135fb3ee-0298-424f-b188-611bdfbec980\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " Nov 03 22:47:46.811123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.810884 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-token\" (UniqueName: \"kubernetes.io/projected/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-token\") pod \"bb5188a0-6554-4eb7-8567-0613709d03d9\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " Nov 03 22:47:46.811123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.810974 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-credential-socket\") pod \"135fb3ee-0298-424f-b188-611bdfbec980\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " Nov 03 22:47:46.811123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.811029 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135fb3ee-0298-424f-b188-611bdfbec980-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135fb3ee-0298-424f-b188-611bdfbec980-istiod-ca-cert\") pod \"135fb3ee-0298-424f-b188-611bdfbec980\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " Nov 03 22:47:46.811123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.811086 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-data\") pod \"bb5188a0-6554-4eb7-8567-0613709d03d9\" (UID: \"bb5188a0-6554-4eb7-8567-0613709d03d9\") " Nov 03 22:47:46.811423 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.811124 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-workload-socket\") pod \"135fb3ee-0298-424f-b188-611bdfbec980\" (UID: \"135fb3ee-0298-424f-b188-611bdfbec980\") " Nov 03 22:47:46.811423 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.811345 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-workload-certs" pod "bb5188a0-6554-4eb7-8567-0613709d03d9" (UID: "bb5188a0-6554-4eb7-8567-0613709d03d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.811423 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.811391 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5188a0-6554-4eb7-8567-0613709d03d9-istiod-ca-cert" pod "bb5188a0-6554-4eb7-8567-0613709d03d9" (UID: "bb5188a0-6554-4eb7-8567-0613709d03d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:46.811874 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.811841 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-data" pod "bb5188a0-6554-4eb7-8567-0613709d03d9" (UID: "bb5188a0-6554-4eb7-8567-0613709d03d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.811990 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.811933 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-workload-certs" pod "135fb3ee-0298-424f-b188-611bdfbec980" (UID: "135fb3ee-0298-424f-b188-611bdfbec980"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.812047 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.812008 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135fb3ee-0298-424f-b188-611bdfbec980-istiod-ca-cert" pod "135fb3ee-0298-424f-b188-611bdfbec980" (UID: "135fb3ee-0298-424f-b188-611bdfbec980"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:46.812191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.812169 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.812266 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.812200 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5188a0-6554-4eb7-8567-0613709d03d9-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.812266 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.812203 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-credential-socket" pod "135fb3ee-0298-424f-b188-611bdfbec980" (UID: "135fb3ee-0298-424f-b188-611bdfbec980"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.812266 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.812215 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.812510 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.812459 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-istio-data" pod "135fb3ee-0298-424f-b188-611bdfbec980" (UID: "135fb3ee-0298-424f-b188-611bdfbec980"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.812608 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.812581 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-workload-socket" pod "135fb3ee-0298-424f-b188-611bdfbec980" (UID: "135fb3ee-0298-424f-b188-611bdfbec980"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.812680 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.812588 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-workload-socket" pod "bb5188a0-6554-4eb7-8567-0613709d03d9" (UID: "bb5188a0-6554-4eb7-8567-0613709d03d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.812680 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.812653 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-credential-socket" pod "bb5188a0-6554-4eb7-8567-0613709d03d9" (UID: "bb5188a0-6554-4eb7-8567-0613709d03d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.813605 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.813577 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135fb3ee-0298-424f-b188-611bdfbec980-kube-api-access-xdg5v" pod "135fb3ee-0298-424f-b188-611bdfbec980" (UID: "135fb3ee-0298-424f-b188-611bdfbec980"). InnerVolumeSpecName "kube-api-access-xdg5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:46.813722 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.813577 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-envoy" pod "bb5188a0-6554-4eb7-8567-0613709d03d9" (UID: "bb5188a0-6554-4eb7-8567-0613709d03d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.813722 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.813675 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-podinfo" pod "bb5188a0-6554-4eb7-8567-0613709d03d9" (UID: "bb5188a0-6554-4eb7-8567-0613709d03d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:46.814221 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.814183 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-istio-envoy" pod "135fb3ee-0298-424f-b188-611bdfbec980" (UID: "135fb3ee-0298-424f-b188-611bdfbec980"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:46.814343 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.814271 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5188a0-6554-4eb7-8567-0613709d03d9-kube-api-access-6w7xz" pod "bb5188a0-6554-4eb7-8567-0613709d03d9" (UID: "bb5188a0-6554-4eb7-8567-0613709d03d9"). InnerVolumeSpecName "kube-api-access-6w7xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:46.814712 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.814691 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-token" pod "bb5188a0-6554-4eb7-8567-0613709d03d9" (UID: "bb5188a0-6554-4eb7-8567-0613709d03d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:46.814807 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.814785 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135fb3ee-0298-424f-b188-611bdfbec980-istio-token" pod "135fb3ee-0298-424f-b188-611bdfbec980" (UID: "135fb3ee-0298-424f-b188-611bdfbec980"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:46.815211 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.815194 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/135fb3ee-0298-424f-b188-611bdfbec980-istio-podinfo" pod "135fb3ee-0298-424f-b188-611bdfbec980" (UID: "135fb3ee-0298-424f-b188-611bdfbec980"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:46.913792 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.913694 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/135fb3ee-0298-424f-b188-611bdfbec980-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.913792 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.913733 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.913792 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.913748 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.913792 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.913768 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.913792 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.913780 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/135fb3ee-0298-424f-b188-611bdfbec980-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.913792 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.913788 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.913792 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.913797 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.914197 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.913806 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.914197 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.913814 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/135fb3ee-0298-424f-b188-611bdfbec980-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.914197 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.913824 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.914197 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.913832 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.914197 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.913841 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/135fb3ee-0298-424f-b188-611bdfbec980-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.914197 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.913862 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w7xz\" (UniqueName: \"kubernetes.io/projected/bb5188a0-6554-4eb7-8567-0613709d03d9-kube-api-access-6w7xz\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.914197 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.913874 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdg5v\" (UniqueName: \"kubernetes.io/projected/135fb3ee-0298-424f-b188-611bdfbec980-kube-api-access-xdg5v\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.914197 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.913884 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5188a0-6554-4eb7-8567-0613709d03d9-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:46.991172 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.991132 5084 generic.go:356] "Generic (PLEG): container finished" podID="bb5188a0-6554-4eb7-8567-0613709d03d9" containerID="fcab45c5e325a7cb1fe394921068dca1d24302e10a213fad173912e8b1a58071" exitCode=0 Nov 03 22:47:46.991377 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.991194 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" event={"ID":"bb5188a0-6554-4eb7-8567-0613709d03d9","Type":"ContainerDied","Data":"fcab45c5e325a7cb1fe394921068dca1d24302e10a213fad173912e8b1a58071"} Nov 03 22:47:46.991377 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.991205 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" Nov 03 22:47:46.991377 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.991261 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm" event={"ID":"bb5188a0-6554-4eb7-8567-0613709d03d9","Type":"ContainerDied","Data":"367869ea2bef77ce2fc88c17757024750506eb011fa8fd11af1414186f875da6"} Nov 03 22:47:46.991377 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.991294 5084 scope.go:122] "RemoveContainer" containerID="fcab45c5e325a7cb1fe394921068dca1d24302e10a213fad173912e8b1a58071" Nov 03 22:47:46.994013 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.993978 5084 generic.go:356] "Generic (PLEG): container finished" podID="135fb3ee-0298-424f-b188-611bdfbec980" containerID="c2dcd4eedb3bcd36b3f68503ec93e2582e62530a7bed4928e11440a3317282d4" exitCode=0 Nov 03 22:47:46.994303 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.994036 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" Nov 03 22:47:46.994303 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.994107 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" event={"ID":"135fb3ee-0298-424f-b188-611bdfbec980","Type":"ContainerDied","Data":"c2dcd4eedb3bcd36b3f68503ec93e2582e62530a7bed4928e11440a3317282d4"} Nov 03 22:47:46.994303 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:46.994161 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7" event={"ID":"135fb3ee-0298-424f-b188-611bdfbec980","Type":"ContainerDied","Data":"1fa5c9f4ae4d6ca6804b903b14e44f0af2e1eeb5635fe1c5b7066c1c3528bba7"} Nov 03 22:47:47.007174 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:47.007142 5084 scope.go:122] "RemoveContainer" containerID="fcab45c5e325a7cb1fe394921068dca1d24302e10a213fad173912e8b1a58071" Nov 03 22:47:47.007667 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:47:47.007637 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcab45c5e325a7cb1fe394921068dca1d24302e10a213fad173912e8b1a58071\": container with ID starting with fcab45c5e325a7cb1fe394921068dca1d24302e10a213fad173912e8b1a58071 not found: ID does not exist" containerID="fcab45c5e325a7cb1fe394921068dca1d24302e10a213fad173912e8b1a58071" Nov 03 22:47:47.007798 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:47.007679 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcab45c5e325a7cb1fe394921068dca1d24302e10a213fad173912e8b1a58071"} err="failed to get container status \"fcab45c5e325a7cb1fe394921068dca1d24302e10a213fad173912e8b1a58071\": rpc error: code = NotFound desc = could not find container \"fcab45c5e325a7cb1fe394921068dca1d24302e10a213fad173912e8b1a58071\": container with ID starting with fcab45c5e325a7cb1fe394921068dca1d24302e10a213fad173912e8b1a58071 not found: ID does not exist" Nov 03 22:47:47.007798 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:47.007706 5084 scope.go:122] "RemoveContainer" containerID="c2dcd4eedb3bcd36b3f68503ec93e2582e62530a7bed4928e11440a3317282d4" Nov 03 22:47:47.024705 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:47.024668 5084 scope.go:122] "RemoveContainer" containerID="e7c7e6f5ad3624692c55b68449e4458e1d2cd6548d6d2b5ac253f3cff07756d9" Nov 03 22:47:47.036910 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:47.036884 5084 scope.go:122] "RemoveContainer" containerID="c2dcd4eedb3bcd36b3f68503ec93e2582e62530a7bed4928e11440a3317282d4" Nov 03 22:47:47.037454 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:47:47.037412 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2dcd4eedb3bcd36b3f68503ec93e2582e62530a7bed4928e11440a3317282d4\": container with ID starting with c2dcd4eedb3bcd36b3f68503ec93e2582e62530a7bed4928e11440a3317282d4 not found: ID does not exist" containerID="c2dcd4eedb3bcd36b3f68503ec93e2582e62530a7bed4928e11440a3317282d4" Nov 03 22:47:47.037588 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:47.037464 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2dcd4eedb3bcd36b3f68503ec93e2582e62530a7bed4928e11440a3317282d4"} err="failed to get container status \"c2dcd4eedb3bcd36b3f68503ec93e2582e62530a7bed4928e11440a3317282d4\": rpc error: code = NotFound desc = could not find container \"c2dcd4eedb3bcd36b3f68503ec93e2582e62530a7bed4928e11440a3317282d4\": container with ID starting with c2dcd4eedb3bcd36b3f68503ec93e2582e62530a7bed4928e11440a3317282d4 not found: ID does not exist" Nov 03 22:47:47.037588 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:47.037492 5084 scope.go:122] "RemoveContainer" containerID="e7c7e6f5ad3624692c55b68449e4458e1d2cd6548d6d2b5ac253f3cff07756d9" Nov 03 22:47:47.037943 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:47:47.037918 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c7e6f5ad3624692c55b68449e4458e1d2cd6548d6d2b5ac253f3cff07756d9\": container with ID starting with e7c7e6f5ad3624692c55b68449e4458e1d2cd6548d6d2b5ac253f3cff07756d9 not found: ID does not exist" containerID="e7c7e6f5ad3624692c55b68449e4458e1d2cd6548d6d2b5ac253f3cff07756d9" Nov 03 22:47:47.038191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:47.037951 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c7e6f5ad3624692c55b68449e4458e1d2cd6548d6d2b5ac253f3cff07756d9"} err="failed to get container status \"e7c7e6f5ad3624692c55b68449e4458e1d2cd6548d6d2b5ac253f3cff07756d9\": rpc error: code = NotFound desc = could not find container \"e7c7e6f5ad3624692c55b68449e4458e1d2cd6548d6d2b5ac253f3cff07756d9\": container with ID starting with e7c7e6f5ad3624692c55b68449e4458e1d2cd6548d6d2b5ac253f3cff07756d9 not found: ID does not exist" Nov 03 22:47:47.232113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:47.231973 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm"] Nov 03 22:47:47.235999 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:47.235959 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-s2qgm"] Nov 03 22:47:47.366429 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:47.366387 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7"] Nov 03 22:47:47.385638 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:47.385596 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3378fd-openshift-default-bb6f586f7-8qmx7"] Nov 03 22:47:48.636941 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.636901 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135fb3ee-0298-424f-b188-611bdfbec980" path="/var/lib/kubelet/pods/135fb3ee-0298-424f-b188-611bdfbec980/volumes" Nov 03 22:47:48.637709 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.637683 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5188a0-6554-4eb7-8567-0613709d03d9" path="/var/lib/kubelet/pods/bb5188a0-6554-4eb7-8567-0613709d03d9/volumes" Nov 03 22:47:48.660700 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.660675 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:47:48.696107 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.696077 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:47:48.745120 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.745085 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/531b6497-408c-4315-806e-0504eead114a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/531b6497-408c-4315-806e-0504eead114a-istio-podinfo\") pod \"531b6497-408c-4315-806e-0504eead114a\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " Nov 03 22:47:48.745333 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.745243 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/531b6497-408c-4315-806e-0504eead114a-kube-api-access-gjj94\" (UniqueName: \"kubernetes.io/projected/531b6497-408c-4315-806e-0504eead114a-kube-api-access-gjj94\") pod \"531b6497-408c-4315-806e-0504eead114a\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " Nov 03 22:47:48.745333 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.745274 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-credential-socket\") pod \"531b6497-408c-4315-806e-0504eead114a\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " Nov 03 22:47:48.745333 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.745309 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/531b6497-408c-4315-806e-0504eead114a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/531b6497-408c-4315-806e-0504eead114a-istiod-ca-cert\") pod \"531b6497-408c-4315-806e-0504eead114a\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " Nov 03 22:47:48.745514 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.745360 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-istio-envoy\") pod \"531b6497-408c-4315-806e-0504eead114a\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " Nov 03 22:47:48.745514 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.745418 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-istio-data\") pod \"531b6497-408c-4315-806e-0504eead114a\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " Nov 03 22:47:48.745514 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.745472 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/531b6497-408c-4315-806e-0504eead114a-istio-token\" (UniqueName: \"kubernetes.io/projected/531b6497-408c-4315-806e-0504eead114a-istio-token\") pod \"531b6497-408c-4315-806e-0504eead114a\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " Nov 03 22:47:48.745669 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.745516 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-workload-socket\") pod \"531b6497-408c-4315-806e-0504eead114a\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " Nov 03 22:47:48.745669 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.745605 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-workload-certs\") pod \"531b6497-408c-4315-806e-0504eead114a\" (UID: \"531b6497-408c-4315-806e-0504eead114a\") " Nov 03 22:47:48.745669 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.745611 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-credential-socket" pod "531b6497-408c-4315-806e-0504eead114a" (UID: "531b6497-408c-4315-806e-0504eead114a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:48.745871 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.745840 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-workload-socket" pod "531b6497-408c-4315-806e-0504eead114a" (UID: "531b6497-408c-4315-806e-0504eead114a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:48.746242 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.745986 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-istio-data" pod "531b6497-408c-4315-806e-0504eead114a" (UID: "531b6497-408c-4315-806e-0504eead114a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:48.746242 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.746054 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-workload-certs" pod "531b6497-408c-4315-806e-0504eead114a" (UID: "531b6497-408c-4315-806e-0504eead114a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:48.746452 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.746279 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/531b6497-408c-4315-806e-0504eead114a-istiod-ca-cert" pod "531b6497-408c-4315-806e-0504eead114a" (UID: "531b6497-408c-4315-806e-0504eead114a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:48.746452 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.746400 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.746452 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.746426 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.746452 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.746439 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.746452 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.746454 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.748604 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.748249 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/531b6497-408c-4315-806e-0504eead114a-istio-podinfo" pod "531b6497-408c-4315-806e-0504eead114a" (UID: "531b6497-408c-4315-806e-0504eead114a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:48.750136 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.748647 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531b6497-408c-4315-806e-0504eead114a-istio-token" pod "531b6497-408c-4315-806e-0504eead114a" (UID: "531b6497-408c-4315-806e-0504eead114a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:48.750136 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.748959 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-istio-envoy" pod "531b6497-408c-4315-806e-0504eead114a" (UID: "531b6497-408c-4315-806e-0504eead114a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:48.750245 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.750220 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531b6497-408c-4315-806e-0504eead114a-kube-api-access-gjj94" pod "531b6497-408c-4315-806e-0504eead114a" (UID: "531b6497-408c-4315-806e-0504eead114a"). InnerVolumeSpecName "kube-api-access-gjj94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:48.847211 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.847168 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-credential-socket\") pod \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " Nov 03 22:47:48.847456 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.847248 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-podinfo\") pod \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " Nov 03 22:47:48.847456 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.847313 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-kube-api-access-rnzbb\" (UniqueName: \"kubernetes.io/projected/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-kube-api-access-rnzbb\") pod \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " Nov 03 22:47:48.847783 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.847742 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-credential-socket" pod "f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" (UID: "f2862fdc-74c6-4f14-9a9b-ced8a3d538d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:48.847969 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.847948 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-envoy\") pod \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " Nov 03 22:47:48.848115 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.848048 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-data\") pod \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " Nov 03 22:47:48.848183 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.848150 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-workload-socket\") pod \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " Nov 03 22:47:48.848250 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.848194 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-token\" (UniqueName: \"kubernetes.io/projected/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-token\") pod \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " Nov 03 22:47:48.848312 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.848253 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-workload-certs\") pod \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " Nov 03 22:47:48.848312 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.848303 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istiod-ca-cert\") pod \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\" (UID: \"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5\") " Nov 03 22:47:48.849057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.848462 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-data" pod "f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" (UID: "f2862fdc-74c6-4f14-9a9b-ced8a3d538d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:48.849057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.848702 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-workload-certs" pod "f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" (UID: "f2862fdc-74c6-4f14-9a9b-ced8a3d538d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:48.849057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.848879 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-workload-socket" pod "f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" (UID: "f2862fdc-74c6-4f14-9a9b-ced8a3d538d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:48.849057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.848958 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/531b6497-408c-4315-806e-0504eead114a-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.849057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.848982 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/531b6497-408c-4315-806e-0504eead114a-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.849057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.848997 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/531b6497-408c-4315-806e-0504eead114a-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.849057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.849012 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.849057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.849024 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istiod-ca-cert" pod "f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" (UID: "f2862fdc-74c6-4f14-9a9b-ced8a3d538d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:48.849057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.849026 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/531b6497-408c-4315-806e-0504eead114a-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.849057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.849053 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.849057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.849082 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.849722 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.849097 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjj94\" (UniqueName: \"kubernetes.io/projected/531b6497-408c-4315-806e-0504eead114a-kube-api-access-gjj94\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.850664 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.850639 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-podinfo" pod "f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" (UID: "f2862fdc-74c6-4f14-9a9b-ced8a3d538d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:48.851436 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.851407 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-token" pod "f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" (UID: "f2862fdc-74c6-4f14-9a9b-ced8a3d538d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:48.852249 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.852227 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-envoy" pod "f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" (UID: "f2862fdc-74c6-4f14-9a9b-ced8a3d538d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:48.852428 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.852395 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-kube-api-access-rnzbb" pod "f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" (UID: "f2862fdc-74c6-4f14-9a9b-ced8a3d538d5"). InnerVolumeSpecName "kube-api-access-rnzbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:48.951554 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.951455 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnzbb\" (UniqueName: \"kubernetes.io/projected/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-kube-api-access-rnzbb\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.951554 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.951499 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.951554 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.951513 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.951554 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.951526 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.951554 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.951540 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:48.951554 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:48.951554 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:47:49.015057 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.015020 5084 generic.go:356] "Generic (PLEG): container finished" podID="531b6497-408c-4315-806e-0504eead114a" containerID="43b1c2e494111edaf6ad2141c39562f40b92641dd0c17ebf3eee031ab3e6d162" exitCode=0 Nov 03 22:47:49.015285 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.015116 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" Nov 03 22:47:49.015285 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.015117 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" event={"ID":"531b6497-408c-4315-806e-0504eead114a","Type":"ContainerDied","Data":"43b1c2e494111edaf6ad2141c39562f40b92641dd0c17ebf3eee031ab3e6d162"} Nov 03 22:47:49.015285 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.015239 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9" event={"ID":"531b6497-408c-4315-806e-0504eead114a","Type":"ContainerDied","Data":"ada9b60eddd3fbcd1383f7d22308232872296d3544ad5ccfb07c45399b904bc6"} Nov 03 22:47:49.015285 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.015264 5084 scope.go:122] "RemoveContainer" containerID="43b1c2e494111edaf6ad2141c39562f40b92641dd0c17ebf3eee031ab3e6d162" Nov 03 22:47:49.017593 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.017556 5084 generic.go:356] "Generic (PLEG): container finished" podID="f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" containerID="e6fa413bffd0164d8e9dda994e28d080418a7b3b598c4a051b8424c7449ad819" exitCode=0 Nov 03 22:47:49.017734 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.017606 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" Nov 03 22:47:49.017734 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.017636 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" event={"ID":"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5","Type":"ContainerDied","Data":"e6fa413bffd0164d8e9dda994e28d080418a7b3b598c4a051b8424c7449ad819"} Nov 03 22:47:49.017734 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.017697 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r" event={"ID":"f2862fdc-74c6-4f14-9a9b-ced8a3d538d5","Type":"ContainerDied","Data":"b2901f46c7231f4b61669c6ac7387c86d26a6e128fbd6f809f12d726606956ed"} Nov 03 22:47:49.027558 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.027534 5084 scope.go:122] "RemoveContainer" containerID="43b1c2e494111edaf6ad2141c39562f40b92641dd0c17ebf3eee031ab3e6d162" Nov 03 22:47:49.027926 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:47:49.027902 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b1c2e494111edaf6ad2141c39562f40b92641dd0c17ebf3eee031ab3e6d162\": container with ID starting with 43b1c2e494111edaf6ad2141c39562f40b92641dd0c17ebf3eee031ab3e6d162 not found: ID does not exist" containerID="43b1c2e494111edaf6ad2141c39562f40b92641dd0c17ebf3eee031ab3e6d162" Nov 03 22:47:49.028013 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.027939 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b1c2e494111edaf6ad2141c39562f40b92641dd0c17ebf3eee031ab3e6d162"} err="failed to get container status \"43b1c2e494111edaf6ad2141c39562f40b92641dd0c17ebf3eee031ab3e6d162\": rpc error: code = NotFound desc = could not find container \"43b1c2e494111edaf6ad2141c39562f40b92641dd0c17ebf3eee031ab3e6d162\": container with ID starting with 43b1c2e494111edaf6ad2141c39562f40b92641dd0c17ebf3eee031ab3e6d162 not found: ID does not exist" Nov 03 22:47:49.028013 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.027978 5084 scope.go:122] "RemoveContainer" containerID="e6fa413bffd0164d8e9dda994e28d080418a7b3b598c4a051b8424c7449ad819" Nov 03 22:47:49.042130 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.042088 5084 scope.go:122] "RemoveContainer" containerID="db198584edb5b9186ad6cf094065137653cab298a0d9687f91c065f47de9793e" Nov 03 22:47:49.053647 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.053620 5084 scope.go:122] "RemoveContainer" containerID="e6fa413bffd0164d8e9dda994e28d080418a7b3b598c4a051b8424c7449ad819" Nov 03 22:47:49.054054 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:47:49.054027 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6fa413bffd0164d8e9dda994e28d080418a7b3b598c4a051b8424c7449ad819\": container with ID starting with e6fa413bffd0164d8e9dda994e28d080418a7b3b598c4a051b8424c7449ad819 not found: ID does not exist" containerID="e6fa413bffd0164d8e9dda994e28d080418a7b3b598c4a051b8424c7449ad819" Nov 03 22:47:49.054162 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.054076 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fa413bffd0164d8e9dda994e28d080418a7b3b598c4a051b8424c7449ad819"} err="failed to get container status \"e6fa413bffd0164d8e9dda994e28d080418a7b3b598c4a051b8424c7449ad819\": rpc error: code = NotFound desc = could not find container \"e6fa413bffd0164d8e9dda994e28d080418a7b3b598c4a051b8424c7449ad819\": container with ID starting with e6fa413bffd0164d8e9dda994e28d080418a7b3b598c4a051b8424c7449ad819 not found: ID does not exist" Nov 03 22:47:49.054162 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.054099 5084 scope.go:122] "RemoveContainer" containerID="db198584edb5b9186ad6cf094065137653cab298a0d9687f91c065f47de9793e" Nov 03 22:47:49.054386 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:47:49.054367 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db198584edb5b9186ad6cf094065137653cab298a0d9687f91c065f47de9793e\": container with ID starting with db198584edb5b9186ad6cf094065137653cab298a0d9687f91c065f47de9793e not found: ID does not exist" containerID="db198584edb5b9186ad6cf094065137653cab298a0d9687f91c065f47de9793e" Nov 03 22:47:49.054425 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.054396 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db198584edb5b9186ad6cf094065137653cab298a0d9687f91c065f47de9793e"} err="failed to get container status \"db198584edb5b9186ad6cf094065137653cab298a0d9687f91c065f47de9793e\": rpc error: code = NotFound desc = could not find container \"db198584edb5b9186ad6cf094065137653cab298a0d9687f91c065f47de9793e\": container with ID starting with db198584edb5b9186ad6cf094065137653cab298a0d9687f91c065f47de9793e not found: ID does not exist" Nov 03 22:47:49.183932 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.183895 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9"] Nov 03 22:47:49.187632 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.187606 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-bdnr9"] Nov 03 22:47:49.298103 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.298039 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r"] Nov 03 22:47:49.301797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:49.301747 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6a6b7f-openshift-default-7d49f446f7-nzv5r"] Nov 03 22:47:50.630685 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:50.630645 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="531b6497-408c-4315-806e-0504eead114a" path="/var/lib/kubelet/pods/531b6497-408c-4315-806e-0504eead114a/volumes" Nov 03 22:47:50.631192 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:47:50.631176 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" path="/var/lib/kubelet/pods/f2862fdc-74c6-4f14-9a9b-ced8a3d538d5/volumes" Nov 03 22:48:12.024828 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:48:12.024786 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:48:42.024600 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:48:42.024560 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:48:44.984451 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:44.984402 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2"] Nov 03 22:48:44.985004 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:44.984703 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" podUID="e533f5a5-fd74-4be2-828b-5efd065720ff" containerName="maas-api" containerID="cri-o://ae498ace186c4b137192f6d5d38495e296acfdc9e543f0458b729d25eef2abf9" gracePeriod=30 Nov 03 22:48:45.325113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.325080 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" Nov 03 22:48:45.422634 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.422595 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e533f5a5-fd74-4be2-828b-5efd065720ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e533f5a5-fd74-4be2-828b-5efd065720ff-metrics-certs\") pod \"e533f5a5-fd74-4be2-828b-5efd065720ff\" (UID: \"e533f5a5-fd74-4be2-828b-5efd065720ff\") " Nov 03 22:48:45.422841 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.422650 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e533f5a5-fd74-4be2-828b-5efd065720ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e533f5a5-fd74-4be2-828b-5efd065720ff-maas-api-tls\") pod \"e533f5a5-fd74-4be2-828b-5efd065720ff\" (UID: \"e533f5a5-fd74-4be2-828b-5efd065720ff\") " Nov 03 22:48:45.422841 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.422721 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e533f5a5-fd74-4be2-828b-5efd065720ff-kube-api-access-mjckn\" (UniqueName: \"kubernetes.io/projected/e533f5a5-fd74-4be2-828b-5efd065720ff-kube-api-access-mjckn\") pod \"e533f5a5-fd74-4be2-828b-5efd065720ff\" (UID: \"e533f5a5-fd74-4be2-828b-5efd065720ff\") " Nov 03 22:48:45.422944 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.422902 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e533f5a5-fd74-4be2-828b-5efd065720ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e533f5a5-fd74-4be2-828b-5efd065720ff-openshift-service-ca\") pod \"e533f5a5-fd74-4be2-828b-5efd065720ff\" (UID: \"e533f5a5-fd74-4be2-828b-5efd065720ff\") " Nov 03 22:48:45.423374 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.423337 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e533f5a5-fd74-4be2-828b-5efd065720ff-openshift-service-ca" pod "e533f5a5-fd74-4be2-828b-5efd065720ff" (UID: "e533f5a5-fd74-4be2-828b-5efd065720ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:45.425145 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.425110 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e533f5a5-fd74-4be2-828b-5efd065720ff-maas-api-tls" pod "e533f5a5-fd74-4be2-828b-5efd065720ff" (UID: "e533f5a5-fd74-4be2-828b-5efd065720ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:48:45.425250 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.425110 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e533f5a5-fd74-4be2-828b-5efd065720ff-kube-api-access-mjckn" pod "e533f5a5-fd74-4be2-828b-5efd065720ff" (UID: "e533f5a5-fd74-4be2-828b-5efd065720ff"). InnerVolumeSpecName "kube-api-access-mjckn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:45.425250 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.425193 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e533f5a5-fd74-4be2-828b-5efd065720ff-metrics-certs" pod "e533f5a5-fd74-4be2-828b-5efd065720ff" (UID: "e533f5a5-fd74-4be2-828b-5efd065720ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:48:45.490899 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.490864 5084 generic.go:356] "Generic (PLEG): container finished" podID="e533f5a5-fd74-4be2-828b-5efd065720ff" containerID="ae498ace186c4b137192f6d5d38495e296acfdc9e543f0458b729d25eef2abf9" exitCode=0 Nov 03 22:48:45.491172 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.490917 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" Nov 03 22:48:45.491285 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.490946 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" event={"ID":"e533f5a5-fd74-4be2-828b-5efd065720ff","Type":"ContainerDied","Data":"ae498ace186c4b137192f6d5d38495e296acfdc9e543f0458b729d25eef2abf9"} Nov 03 22:48:45.491353 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.491300 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2" event={"ID":"e533f5a5-fd74-4be2-828b-5efd065720ff","Type":"ContainerDied","Data":"2bc4bad07363e30021d2ff8f881a4f7040149e51435a42e39e704f1f67b0b031"} Nov 03 22:48:45.491353 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.491338 5084 scope.go:122] "RemoveContainer" containerID="ae498ace186c4b137192f6d5d38495e296acfdc9e543f0458b729d25eef2abf9" Nov 03 22:48:45.504325 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.504295 5084 scope.go:122] "RemoveContainer" containerID="ae498ace186c4b137192f6d5d38495e296acfdc9e543f0458b729d25eef2abf9" Nov 03 22:48:45.504693 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:48:45.504668 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae498ace186c4b137192f6d5d38495e296acfdc9e543f0458b729d25eef2abf9\": container with ID starting with ae498ace186c4b137192f6d5d38495e296acfdc9e543f0458b729d25eef2abf9 not found: ID does not exist" containerID="ae498ace186c4b137192f6d5d38495e296acfdc9e543f0458b729d25eef2abf9" Nov 03 22:48:45.504784 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.504706 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae498ace186c4b137192f6d5d38495e296acfdc9e543f0458b729d25eef2abf9"} err="failed to get container status \"ae498ace186c4b137192f6d5d38495e296acfdc9e543f0458b729d25eef2abf9\": rpc error: code = NotFound desc = could not find container \"ae498ace186c4b137192f6d5d38495e296acfdc9e543f0458b729d25eef2abf9\": container with ID starting with ae498ace186c4b137192f6d5d38495e296acfdc9e543f0458b729d25eef2abf9 not found: ID does not exist" Nov 03 22:48:45.524707 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.524672 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjckn\" (UniqueName: \"kubernetes.io/projected/e533f5a5-fd74-4be2-828b-5efd065720ff-kube-api-access-mjckn\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:45.524707 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.524704 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e533f5a5-fd74-4be2-828b-5efd065720ff-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:45.524914 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.524719 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e533f5a5-fd74-4be2-828b-5efd065720ff-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:45.524914 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.524728 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e533f5a5-fd74-4be2-828b-5efd065720ff-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:48:45.663413 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.663305 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2"] Nov 03 22:48:45.670963 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:45.670916 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f881e3-64c55b45b5-8rbf2"] Nov 03 22:48:46.630987 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:48:46.630918 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e533f5a5-fd74-4be2-828b-5efd065720ff" path="/var/lib/kubelet/pods/e533f5a5-fd74-4be2-828b-5efd065720ff/volumes" Nov 03 22:49:12.026131 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:49:12.026091 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:49:42.023912 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:49:42.023866 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:50:05.398488 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:05.398444 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2"] Nov 03 22:50:05.399132 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:05.398718 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" podUID="ae836564-7d17-4919-8a65-f655fabe9759" containerName="maas-api" containerID="cri-o://f68420503c9bc5a9672d09ab7f5e77c560f61f5b014c5e88c052591a6ddc9088" gracePeriod=30 Nov 03 22:50:05.705747 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:05.705714 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:50:05.775976 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:05.775926 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae836564-7d17-4919-8a65-f655fabe9759-kube-api-access-6w9mp\" (UniqueName: \"kubernetes.io/projected/ae836564-7d17-4919-8a65-f655fabe9759-kube-api-access-6w9mp\") pod \"ae836564-7d17-4919-8a65-f655fabe9759\" (UID: \"ae836564-7d17-4919-8a65-f655fabe9759\") " Nov 03 22:50:05.776243 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:05.776108 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae836564-7d17-4919-8a65-f655fabe9759-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae836564-7d17-4919-8a65-f655fabe9759-maas-api-tls\") pod \"ae836564-7d17-4919-8a65-f655fabe9759\" (UID: \"ae836564-7d17-4919-8a65-f655fabe9759\") " Nov 03 22:50:05.776325 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:05.776303 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae836564-7d17-4919-8a65-f655fabe9759-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae836564-7d17-4919-8a65-f655fabe9759-openshift-service-ca\") pod \"ae836564-7d17-4919-8a65-f655fabe9759\" (UID: \"ae836564-7d17-4919-8a65-f655fabe9759\") " Nov 03 22:50:05.776389 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:05.776364 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae836564-7d17-4919-8a65-f655fabe9759-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae836564-7d17-4919-8a65-f655fabe9759-metrics-certs\") pod \"ae836564-7d17-4919-8a65-f655fabe9759\" (UID: \"ae836564-7d17-4919-8a65-f655fabe9759\") " Nov 03 22:50:05.776620 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:05.776588 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae836564-7d17-4919-8a65-f655fabe9759-openshift-service-ca" pod "ae836564-7d17-4919-8a65-f655fabe9759" (UID: "ae836564-7d17-4919-8a65-f655fabe9759"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:05.776901 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:05.776877 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae836564-7d17-4919-8a65-f655fabe9759-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:05.778284 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:05.778256 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae836564-7d17-4919-8a65-f655fabe9759-maas-api-tls" pod "ae836564-7d17-4919-8a65-f655fabe9759" (UID: "ae836564-7d17-4919-8a65-f655fabe9759"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:05.778387 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:05.778324 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae836564-7d17-4919-8a65-f655fabe9759-kube-api-access-6w9mp" pod "ae836564-7d17-4919-8a65-f655fabe9759" (UID: "ae836564-7d17-4919-8a65-f655fabe9759"). InnerVolumeSpecName "kube-api-access-6w9mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:05.778433 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:05.778400 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae836564-7d17-4919-8a65-f655fabe9759-metrics-certs" pod "ae836564-7d17-4919-8a65-f655fabe9759" (UID: "ae836564-7d17-4919-8a65-f655fabe9759"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:05.877936 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:05.877902 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae836564-7d17-4919-8a65-f655fabe9759-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:05.877936 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:05.877934 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w9mp\" (UniqueName: \"kubernetes.io/projected/ae836564-7d17-4919-8a65-f655fabe9759-kube-api-access-6w9mp\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:05.878195 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:05.877947 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae836564-7d17-4919-8a65-f655fabe9759-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:06.141449 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:06.141406 5084 generic.go:356] "Generic (PLEG): container finished" podID="ae836564-7d17-4919-8a65-f655fabe9759" containerID="f68420503c9bc5a9672d09ab7f5e77c560f61f5b014c5e88c052591a6ddc9088" exitCode=0 Nov 03 22:50:06.141655 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:06.141453 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" Nov 03 22:50:06.141655 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:06.141489 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" event={"ID":"ae836564-7d17-4919-8a65-f655fabe9759","Type":"ContainerDied","Data":"f68420503c9bc5a9672d09ab7f5e77c560f61f5b014c5e88c052591a6ddc9088"} Nov 03 22:50:06.141655 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:06.141544 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2" event={"ID":"ae836564-7d17-4919-8a65-f655fabe9759","Type":"ContainerDied","Data":"cb177c14d74f96556f5dad48597b57e08780125e0b2c721a1be7de8f28f39f54"} Nov 03 22:50:06.141655 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:06.141566 5084 scope.go:122] "RemoveContainer" containerID="f68420503c9bc5a9672d09ab7f5e77c560f61f5b014c5e88c052591a6ddc9088" Nov 03 22:50:06.154100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:06.154070 5084 scope.go:122] "RemoveContainer" containerID="f68420503c9bc5a9672d09ab7f5e77c560f61f5b014c5e88c052591a6ddc9088" Nov 03 22:50:06.154438 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:50:06.154413 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68420503c9bc5a9672d09ab7f5e77c560f61f5b014c5e88c052591a6ddc9088\": container with ID starting with f68420503c9bc5a9672d09ab7f5e77c560f61f5b014c5e88c052591a6ddc9088 not found: ID does not exist" containerID="f68420503c9bc5a9672d09ab7f5e77c560f61f5b014c5e88c052591a6ddc9088" Nov 03 22:50:06.154514 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:06.154446 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68420503c9bc5a9672d09ab7f5e77c560f61f5b014c5e88c052591a6ddc9088"} err="failed to get container status \"f68420503c9bc5a9672d09ab7f5e77c560f61f5b014c5e88c052591a6ddc9088\": rpc error: code = NotFound desc = could not find container \"f68420503c9bc5a9672d09ab7f5e77c560f61f5b014c5e88c052591a6ddc9088\": container with ID starting with f68420503c9bc5a9672d09ab7f5e77c560f61f5b014c5e88c052591a6ddc9088 not found: ID does not exist" Nov 03 22:50:06.310388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:06.310356 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2"] Nov 03 22:50:06.313952 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:06.313918 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-01bb3a-74f745d948-rbqq2"] Nov 03 22:50:06.630425 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:06.630383 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae836564-7d17-4919-8a65-f655fabe9759" path="/var/lib/kubelet/pods/ae836564-7d17-4919-8a65-f655fabe9759/volumes" Nov 03 22:50:12.024997 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:50:12.024958 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:50:19.709183 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:19.709138 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn"] Nov 03 22:50:25.998275 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:25.998239 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr"] Nov 03 22:50:30.998759 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:30.998679 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr" podUID="9fbbe6f8-7037-43d3-b3e3-0e48b6316632" containerName="payload-processing" containerID="cri-o://75f3fa604b2e6c35d734eb58327c129665c28e2ed936a1e9ffabcc98da99bd17" gracePeriod=25 Nov 03 22:50:31.293834 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:31.293798 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr" Nov 03 22:50:31.351983 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:31.351938 5084 generic.go:356] "Generic (PLEG): container finished" podID="9fbbe6f8-7037-43d3-b3e3-0e48b6316632" containerID="75f3fa604b2e6c35d734eb58327c129665c28e2ed936a1e9ffabcc98da99bd17" exitCode=0 Nov 03 22:50:31.352222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:31.352013 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr" Nov 03 22:50:31.352222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:31.352028 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr" event={"ID":"9fbbe6f8-7037-43d3-b3e3-0e48b6316632","Type":"ContainerDied","Data":"75f3fa604b2e6c35d734eb58327c129665c28e2ed936a1e9ffabcc98da99bd17"} Nov 03 22:50:31.352222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:31.352152 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr" event={"ID":"9fbbe6f8-7037-43d3-b3e3-0e48b6316632","Type":"ContainerDied","Data":"5ae596a15ea1c6f8ab9a49fc95428fd4bb7353402e674e347ec2faf85d8f699e"} Nov 03 22:50:31.352222 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:31.352168 5084 scope.go:122] "RemoveContainer" containerID="75f3fa604b2e6c35d734eb58327c129665c28e2ed936a1e9ffabcc98da99bd17" Nov 03 22:50:31.363204 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:31.363178 5084 scope.go:122] "RemoveContainer" containerID="75f3fa604b2e6c35d734eb58327c129665c28e2ed936a1e9ffabcc98da99bd17" Nov 03 22:50:31.363581 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:50:31.363547 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f3fa604b2e6c35d734eb58327c129665c28e2ed936a1e9ffabcc98da99bd17\": container with ID starting with 75f3fa604b2e6c35d734eb58327c129665c28e2ed936a1e9ffabcc98da99bd17 not found: ID does not exist" containerID="75f3fa604b2e6c35d734eb58327c129665c28e2ed936a1e9ffabcc98da99bd17" Nov 03 22:50:31.363664 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:31.363594 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f3fa604b2e6c35d734eb58327c129665c28e2ed936a1e9ffabcc98da99bd17"} err="failed to get container status \"75f3fa604b2e6c35d734eb58327c129665c28e2ed936a1e9ffabcc98da99bd17\": rpc error: code = NotFound desc = could not find container \"75f3fa604b2e6c35d734eb58327c129665c28e2ed936a1e9ffabcc98da99bd17\": container with ID starting with 75f3fa604b2e6c35d734eb58327c129665c28e2ed936a1e9ffabcc98da99bd17 not found: ID does not exist" Nov 03 22:50:31.473368 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:31.473328 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbbe6f8-7037-43d3-b3e3-0e48b6316632-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbbe6f8-7037-43d3-b3e3-0e48b6316632-plugins-config-volume\") pod \"9fbbe6f8-7037-43d3-b3e3-0e48b6316632\" (UID: \"9fbbe6f8-7037-43d3-b3e3-0e48b6316632\") " Nov 03 22:50:31.473649 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:31.473417 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbbe6f8-7037-43d3-b3e3-0e48b6316632-kube-api-access-jm2nx\" (UniqueName: \"kubernetes.io/projected/9fbbe6f8-7037-43d3-b3e3-0e48b6316632-kube-api-access-jm2nx\") pod \"9fbbe6f8-7037-43d3-b3e3-0e48b6316632\" (UID: \"9fbbe6f8-7037-43d3-b3e3-0e48b6316632\") " Nov 03 22:50:31.473814 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:31.473754 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbbe6f8-7037-43d3-b3e3-0e48b6316632-plugins-config-volume" pod "9fbbe6f8-7037-43d3-b3e3-0e48b6316632" (UID: "9fbbe6f8-7037-43d3-b3e3-0e48b6316632"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:31.473966 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:31.473944 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbbe6f8-7037-43d3-b3e3-0e48b6316632-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:31.475701 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:31.475672 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbbe6f8-7037-43d3-b3e3-0e48b6316632-kube-api-access-jm2nx" pod "9fbbe6f8-7037-43d3-b3e3-0e48b6316632" (UID: "9fbbe6f8-7037-43d3-b3e3-0e48b6316632"). InnerVolumeSpecName "kube-api-access-jm2nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:31.575724 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:31.575680 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm2nx\" (UniqueName: \"kubernetes.io/projected/9fbbe6f8-7037-43d3-b3e3-0e48b6316632-kube-api-access-jm2nx\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:31.778441 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:31.778405 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr"] Nov 03 22:50:31.829098 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:31.829005 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f61714-7ff745db8-ttsnr"] Nov 03 22:50:32.628697 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:32.628661 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbbe6f8-7037-43d3-b3e3-0e48b6316632" path="/var/lib/kubelet/pods/9fbbe6f8-7037-43d3-b3e3-0e48b6316632/volumes" Nov 03 22:50:32.940756 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:32.940666 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9"] Nov 03 22:50:34.733144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:34.733103 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" containerID="cri-o://063b182003e3bb3a31fb2180b042ff1921530c892a199257dea6b73e4e3124b7" gracePeriod=45 Nov 03 22:50:35.033119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.033084 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:50:35.225142 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.225094 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b28118-7e39-4b66-be5f-f5a61a199f85-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b28118-7e39-4b66-be5f-f5a61a199f85-tls-certs\") pod \"a8b28118-7e39-4b66-be5f-f5a61a199f85\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " Nov 03 22:50:35.225342 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.225220 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-home\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-home\") pod \"a8b28118-7e39-4b66-be5f-f5a61a199f85\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " Nov 03 22:50:35.225342 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.225244 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-dshm\") pod \"a8b28118-7e39-4b66-be5f-f5a61a199f85\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " Nov 03 22:50:35.225342 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.225301 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b28118-7e39-4b66-be5f-f5a61a199f85-kube-api-access-v8t6t\" (UniqueName: \"kubernetes.io/projected/a8b28118-7e39-4b66-be5f-f5a61a199f85-kube-api-access-v8t6t\") pod \"a8b28118-7e39-4b66-be5f-f5a61a199f85\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " Nov 03 22:50:35.225488 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.225366 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-model-cache\") pod \"a8b28118-7e39-4b66-be5f-f5a61a199f85\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " Nov 03 22:50:35.225488 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.225479 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-tmp-dir\") pod \"a8b28118-7e39-4b66-be5f-f5a61a199f85\" (UID: \"a8b28118-7e39-4b66-be5f-f5a61a199f85\") " Nov 03 22:50:35.225600 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.225567 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-home" pod "a8b28118-7e39-4b66-be5f-f5a61a199f85" (UID: "a8b28118-7e39-4b66-be5f-f5a61a199f85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.225739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.225704 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-model-cache" pod "a8b28118-7e39-4b66-be5f-f5a61a199f85" (UID: "a8b28118-7e39-4b66-be5f-f5a61a199f85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.225940 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.225827 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-tmp-dir" pod "a8b28118-7e39-4b66-be5f-f5a61a199f85" (UID: "a8b28118-7e39-4b66-be5f-f5a61a199f85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.226161 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.226103 5084 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-model-cache\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.226161 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.226138 5084 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-tmp-dir\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.226161 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.226153 5084 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-home\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.227713 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.227684 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-dshm" pod "a8b28118-7e39-4b66-be5f-f5a61a199f85" (UID: "a8b28118-7e39-4b66-be5f-f5a61a199f85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:35.227819 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.227723 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b28118-7e39-4b66-be5f-f5a61a199f85-tls-certs" pod "a8b28118-7e39-4b66-be5f-f5a61a199f85" (UID: "a8b28118-7e39-4b66-be5f-f5a61a199f85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:35.227819 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.227772 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b28118-7e39-4b66-be5f-f5a61a199f85-kube-api-access-v8t6t" pod "a8b28118-7e39-4b66-be5f-f5a61a199f85" (UID: "a8b28118-7e39-4b66-be5f-f5a61a199f85"). InnerVolumeSpecName "kube-api-access-v8t6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:35.330954 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.330000 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8t6t\" (UniqueName: \"kubernetes.io/projected/a8b28118-7e39-4b66-be5f-f5a61a199f85-kube-api-access-v8t6t\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.330954 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.330079 5084 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b28118-7e39-4b66-be5f-f5a61a199f85-tls-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.330954 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.330097 5084 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8b28118-7e39-4b66-be5f-f5a61a199f85-dshm\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:35.392768 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.392727 5084 generic.go:356] "Generic (PLEG): container finished" podID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerID="063b182003e3bb3a31fb2180b042ff1921530c892a199257dea6b73e4e3124b7" exitCode=0 Nov 03 22:50:35.392961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.392788 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" event={"ID":"a8b28118-7e39-4b66-be5f-f5a61a199f85","Type":"ContainerDied","Data":"063b182003e3bb3a31fb2180b042ff1921530c892a199257dea6b73e4e3124b7"} Nov 03 22:50:35.392961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.392806 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" Nov 03 22:50:35.392961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.392832 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn" event={"ID":"a8b28118-7e39-4b66-be5f-f5a61a199f85","Type":"ContainerDied","Data":"fb3f8e14d3036b97b0d5096c9d6cd1421d383803cf5d4f8ade17572a2a41d898"} Nov 03 22:50:35.392961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.392856 5084 scope.go:122] "RemoveContainer" containerID="063b182003e3bb3a31fb2180b042ff1921530c892a199257dea6b73e4e3124b7" Nov 03 22:50:35.405694 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.405662 5084 scope.go:122] "RemoveContainer" containerID="063b182003e3bb3a31fb2180b042ff1921530c892a199257dea6b73e4e3124b7" Nov 03 22:50:35.406090 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:50:35.406046 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"063b182003e3bb3a31fb2180b042ff1921530c892a199257dea6b73e4e3124b7\": container with ID starting with 063b182003e3bb3a31fb2180b042ff1921530c892a199257dea6b73e4e3124b7 not found: ID does not exist" containerID="063b182003e3bb3a31fb2180b042ff1921530c892a199257dea6b73e4e3124b7" Nov 03 22:50:35.406173 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.406103 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"063b182003e3bb3a31fb2180b042ff1921530c892a199257dea6b73e4e3124b7"} err="failed to get container status \"063b182003e3bb3a31fb2180b042ff1921530c892a199257dea6b73e4e3124b7\": rpc error: code = NotFound desc = could not find container \"063b182003e3bb3a31fb2180b042ff1921530c892a199257dea6b73e4e3124b7\": container with ID starting with 063b182003e3bb3a31fb2180b042ff1921530c892a199257dea6b73e4e3124b7 not found: ID does not exist" Nov 03 22:50:35.562192 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.562148 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn"] Nov 03 22:50:35.565707 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:35.565671 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f61714/distinct-w0-f61714-kserve-7655b788d8-p8kqn"] Nov 03 22:50:36.629182 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:36.629143 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" path="/var/lib/kubelet/pods/a8b28118-7e39-4b66-be5f-f5a61a199f85/volumes" Nov 03 22:50:36.659754 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:36.659713 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll"] Nov 03 22:50:36.660023 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:36.659974 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" podUID="2da4671a-8a4b-4ce4-9eb7-81183fe89ad6" containerName="maas-api" containerID="cri-o://ecc7bed8828154a584ce4b43f9362496923293560a32bedb2c55209391232874" gracePeriod=30 Nov 03 22:50:37.021553 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.021524 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:50:37.151681 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.151638 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-openshift-service-ca\") pod \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\" (UID: \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\") " Nov 03 22:50:37.151890 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.151776 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-kube-api-access-9k84h\" (UniqueName: \"kubernetes.io/projected/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-kube-api-access-9k84h\") pod \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\" (UID: \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\") " Nov 03 22:50:37.151890 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.151849 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-metrics-certs\") pod \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\" (UID: \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\") " Nov 03 22:50:37.152023 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.151890 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-maas-api-tls\") pod \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\" (UID: \"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6\") " Nov 03 22:50:37.152100 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.152083 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-openshift-service-ca" pod "2da4671a-8a4b-4ce4-9eb7-81183fe89ad6" (UID: "2da4671a-8a4b-4ce4-9eb7-81183fe89ad6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:37.152456 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.152431 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:37.154823 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.154786 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-metrics-certs" pod "2da4671a-8a4b-4ce4-9eb7-81183fe89ad6" (UID: "2da4671a-8a4b-4ce4-9eb7-81183fe89ad6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:37.156131 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.155233 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-kube-api-access-9k84h" pod "2da4671a-8a4b-4ce4-9eb7-81183fe89ad6" (UID: "2da4671a-8a4b-4ce4-9eb7-81183fe89ad6"). InnerVolumeSpecName "kube-api-access-9k84h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:37.156131 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.155674 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-maas-api-tls" pod "2da4671a-8a4b-4ce4-9eb7-81183fe89ad6" (UID: "2da4671a-8a4b-4ce4-9eb7-81183fe89ad6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:37.254471 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.254300 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k84h\" (UniqueName: \"kubernetes.io/projected/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-kube-api-access-9k84h\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:37.254471 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.254374 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:37.254471 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.254388 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:37.410623 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.410583 5084 generic.go:356] "Generic (PLEG): container finished" podID="2da4671a-8a4b-4ce4-9eb7-81183fe89ad6" containerID="ecc7bed8828154a584ce4b43f9362496923293560a32bedb2c55209391232874" exitCode=0 Nov 03 22:50:37.410859 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.410688 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" event={"ID":"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6","Type":"ContainerDied","Data":"ecc7bed8828154a584ce4b43f9362496923293560a32bedb2c55209391232874"} Nov 03 22:50:37.410859 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.410699 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" Nov 03 22:50:37.410859 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.410745 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll" event={"ID":"2da4671a-8a4b-4ce4-9eb7-81183fe89ad6","Type":"ContainerDied","Data":"1f7a65fecb994e3b38411f16f1fa774f00464eb08f53bbbb5b6bc9e35ade513e"} Nov 03 22:50:37.410859 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.410775 5084 scope.go:122] "RemoveContainer" containerID="ecc7bed8828154a584ce4b43f9362496923293560a32bedb2c55209391232874" Nov 03 22:50:37.426534 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.426506 5084 scope.go:122] "RemoveContainer" containerID="ecc7bed8828154a584ce4b43f9362496923293560a32bedb2c55209391232874" Nov 03 22:50:37.426909 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:50:37.426886 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc7bed8828154a584ce4b43f9362496923293560a32bedb2c55209391232874\": container with ID starting with ecc7bed8828154a584ce4b43f9362496923293560a32bedb2c55209391232874 not found: ID does not exist" containerID="ecc7bed8828154a584ce4b43f9362496923293560a32bedb2c55209391232874" Nov 03 22:50:37.426965 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.426920 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc7bed8828154a584ce4b43f9362496923293560a32bedb2c55209391232874"} err="failed to get container status \"ecc7bed8828154a584ce4b43f9362496923293560a32bedb2c55209391232874\": rpc error: code = NotFound desc = could not find container \"ecc7bed8828154a584ce4b43f9362496923293560a32bedb2c55209391232874\": container with ID starting with ecc7bed8828154a584ce4b43f9362496923293560a32bedb2c55209391232874 not found: ID does not exist" Nov 03 22:50:37.581569 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.581532 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll"] Nov 03 22:50:37.587252 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:37.587212 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1b2fb1-6769977b59-qh9ll"] Nov 03 22:50:38.629793 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:38.629745 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da4671a-8a4b-4ce4-9eb7-81183fe89ad6" path="/var/lib/kubelet/pods/2da4671a-8a4b-4ce4-9eb7-81183fe89ad6/volumes" Nov 03 22:50:42.023827 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:50:42.023786 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:50:43.017708 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.017669 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h"] Nov 03 22:50:43.017950 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.017908 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" podUID="e76224de-abc4-45e7-8502-87f35e73f3cb" containerName="maas-api" containerID="cri-o://ab2d5cffd8f926ef366e82b07c42a392b574e72e13ddc0ac846e90cfdd839ad0" gracePeriod=30 Nov 03 22:50:43.318451 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.318415 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:50:43.446263 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.446185 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e76224de-abc4-45e7-8502-87f35e73f3cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e76224de-abc4-45e7-8502-87f35e73f3cb-maas-api-tls\") pod \"e76224de-abc4-45e7-8502-87f35e73f3cb\" (UID: \"e76224de-abc4-45e7-8502-87f35e73f3cb\") " Nov 03 22:50:43.446755 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.446718 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e76224de-abc4-45e7-8502-87f35e73f3cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e76224de-abc4-45e7-8502-87f35e73f3cb-metrics-certs\") pod \"e76224de-abc4-45e7-8502-87f35e73f3cb\" (UID: \"e76224de-abc4-45e7-8502-87f35e73f3cb\") " Nov 03 22:50:43.462268 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.460608 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e76224de-abc4-45e7-8502-87f35e73f3cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e76224de-abc4-45e7-8502-87f35e73f3cb-openshift-service-ca\") pod \"e76224de-abc4-45e7-8502-87f35e73f3cb\" (UID: \"e76224de-abc4-45e7-8502-87f35e73f3cb\") " Nov 03 22:50:43.462268 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.460697 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76224de-abc4-45e7-8502-87f35e73f3cb-kube-api-access-mm6cf\" (UniqueName: \"kubernetes.io/projected/e76224de-abc4-45e7-8502-87f35e73f3cb-kube-api-access-mm6cf\") pod \"e76224de-abc4-45e7-8502-87f35e73f3cb\" (UID: \"e76224de-abc4-45e7-8502-87f35e73f3cb\") " Nov 03 22:50:43.462552 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.462498 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76224de-abc4-45e7-8502-87f35e73f3cb-openshift-service-ca" pod "e76224de-abc4-45e7-8502-87f35e73f3cb" (UID: "e76224de-abc4-45e7-8502-87f35e73f3cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:43.464669 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.464456 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e76224de-abc4-45e7-8502-87f35e73f3cb-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:43.470917 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.469385 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e76224de-abc4-45e7-8502-87f35e73f3cb-metrics-certs" pod "e76224de-abc4-45e7-8502-87f35e73f3cb" (UID: "e76224de-abc4-45e7-8502-87f35e73f3cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:43.470917 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.470188 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76224de-abc4-45e7-8502-87f35e73f3cb-kube-api-access-mm6cf" pod "e76224de-abc4-45e7-8502-87f35e73f3cb" (UID: "e76224de-abc4-45e7-8502-87f35e73f3cb"). InnerVolumeSpecName "kube-api-access-mm6cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:43.470917 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.470855 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e76224de-abc4-45e7-8502-87f35e73f3cb-maas-api-tls" pod "e76224de-abc4-45e7-8502-87f35e73f3cb" (UID: "e76224de-abc4-45e7-8502-87f35e73f3cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:43.503714 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.503670 5084 generic.go:356] "Generic (PLEG): container finished" podID="e76224de-abc4-45e7-8502-87f35e73f3cb" containerID="ab2d5cffd8f926ef366e82b07c42a392b574e72e13ddc0ac846e90cfdd839ad0" exitCode=0 Nov 03 22:50:43.503917 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.503732 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" event={"ID":"e76224de-abc4-45e7-8502-87f35e73f3cb","Type":"ContainerDied","Data":"ab2d5cffd8f926ef366e82b07c42a392b574e72e13ddc0ac846e90cfdd839ad0"} Nov 03 22:50:43.503917 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.503764 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" Nov 03 22:50:43.503917 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.503795 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h" event={"ID":"e76224de-abc4-45e7-8502-87f35e73f3cb","Type":"ContainerDied","Data":"556f6d81fa7779155897580bd27bafc94cce9bbbef960ae7fec287f6c3956aaa"} Nov 03 22:50:43.503917 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.503833 5084 scope.go:122] "RemoveContainer" containerID="ab2d5cffd8f926ef366e82b07c42a392b574e72e13ddc0ac846e90cfdd839ad0" Nov 03 22:50:43.521132 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.521046 5084 scope.go:122] "RemoveContainer" containerID="ab2d5cffd8f926ef366e82b07c42a392b574e72e13ddc0ac846e90cfdd839ad0" Nov 03 22:50:43.521520 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:50:43.521500 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2d5cffd8f926ef366e82b07c42a392b574e72e13ddc0ac846e90cfdd839ad0\": container with ID starting with ab2d5cffd8f926ef366e82b07c42a392b574e72e13ddc0ac846e90cfdd839ad0 not found: ID does not exist" containerID="ab2d5cffd8f926ef366e82b07c42a392b574e72e13ddc0ac846e90cfdd839ad0" Nov 03 22:50:43.521635 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.521530 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2d5cffd8f926ef366e82b07c42a392b574e72e13ddc0ac846e90cfdd839ad0"} err="failed to get container status \"ab2d5cffd8f926ef366e82b07c42a392b574e72e13ddc0ac846e90cfdd839ad0\": rpc error: code = NotFound desc = could not find container \"ab2d5cffd8f926ef366e82b07c42a392b574e72e13ddc0ac846e90cfdd839ad0\": container with ID starting with ab2d5cffd8f926ef366e82b07c42a392b574e72e13ddc0ac846e90cfdd839ad0 not found: ID does not exist" Nov 03 22:50:43.584151 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.581990 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e76224de-abc4-45e7-8502-87f35e73f3cb-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:43.584151 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.582032 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm6cf\" (UniqueName: \"kubernetes.io/projected/e76224de-abc4-45e7-8502-87f35e73f3cb-kube-api-access-mm6cf\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:43.584151 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.582186 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e76224de-abc4-45e7-8502-87f35e73f3cb-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:43.697790 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.697715 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h"] Nov 03 22:50:43.701741 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:43.701711 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8f168-5c96675985-d958h"] Nov 03 22:50:44.629692 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:44.629653 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76224de-abc4-45e7-8502-87f35e73f3cb" path="/var/lib/kubelet/pods/e76224de-abc4-45e7-8502-87f35e73f3cb/volumes" Nov 03 22:50:47.964715 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:47.964668 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" podUID="df48770c-9c54-43d0-87c8-bc7a65bebf2c" containerName="main" containerID="cri-o://6a1e70a0d8fd1d73f15f9f2001698d45c1d4ae8b13a61279c46492aee43ef6de" gracePeriod=45 Nov 03 22:50:48.254976 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.254944 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:50:48.343719 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.343682 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-tmp-dir\") pod \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " Nov 03 22:50:48.344031 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.343735 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df48770c-9c54-43d0-87c8-bc7a65bebf2c-kube-api-access-7j2sx\" (UniqueName: \"kubernetes.io/projected/df48770c-9c54-43d0-87c8-bc7a65bebf2c-kube-api-access-7j2sx\") pod \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " Nov 03 22:50:48.344031 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.343817 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-dshm\") pod \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " Nov 03 22:50:48.344031 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.343858 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-home\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-home\") pod \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " Nov 03 22:50:48.344031 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.343901 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-model-cache\") pod \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " Nov 03 22:50:48.344031 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.343936 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df48770c-9c54-43d0-87c8-bc7a65bebf2c-tls-certs\" (UniqueName: \"kubernetes.io/secret/df48770c-9c54-43d0-87c8-bc7a65bebf2c-tls-certs\") pod \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\" (UID: \"df48770c-9c54-43d0-87c8-bc7a65bebf2c\") " Nov 03 22:50:48.344434 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.344037 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-tmp-dir" pod "df48770c-9c54-43d0-87c8-bc7a65bebf2c" (UID: "df48770c-9c54-43d0-87c8-bc7a65bebf2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:48.344434 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.344351 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-home" pod "df48770c-9c54-43d0-87c8-bc7a65bebf2c" (UID: "df48770c-9c54-43d0-87c8-bc7a65bebf2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:48.344434 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.344361 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-model-cache" pod "df48770c-9c54-43d0-87c8-bc7a65bebf2c" (UID: "df48770c-9c54-43d0-87c8-bc7a65bebf2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:48.344601 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.344539 5084 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-tmp-dir\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:48.344601 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.344558 5084 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-home\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:48.344601 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.344570 5084 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-model-cache\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:48.346286 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.346257 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-dshm" pod "df48770c-9c54-43d0-87c8-bc7a65bebf2c" (UID: "df48770c-9c54-43d0-87c8-bc7a65bebf2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:48.346387 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.346372 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df48770c-9c54-43d0-87c8-bc7a65bebf2c-tls-certs" pod "df48770c-9c54-43d0-87c8-bc7a65bebf2c" (UID: "df48770c-9c54-43d0-87c8-bc7a65bebf2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:50:48.346455 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.346417 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df48770c-9c54-43d0-87c8-bc7a65bebf2c-kube-api-access-7j2sx" pod "df48770c-9c54-43d0-87c8-bc7a65bebf2c" (UID: "df48770c-9c54-43d0-87c8-bc7a65bebf2c"). InnerVolumeSpecName "kube-api-access-7j2sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:48.445693 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.445645 5084 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df48770c-9c54-43d0-87c8-bc7a65bebf2c-tls-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:48.445693 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.445683 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j2sx\" (UniqueName: \"kubernetes.io/projected/df48770c-9c54-43d0-87c8-bc7a65bebf2c-kube-api-access-7j2sx\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:48.445693 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.445699 5084 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df48770c-9c54-43d0-87c8-bc7a65bebf2c-dshm\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:50:48.558123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.558088 5084 generic.go:356] "Generic (PLEG): container finished" podID="df48770c-9c54-43d0-87c8-bc7a65bebf2c" containerID="6a1e70a0d8fd1d73f15f9f2001698d45c1d4ae8b13a61279c46492aee43ef6de" exitCode=0 Nov 03 22:50:48.558123 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.558115 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" Nov 03 22:50:48.558436 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.558154 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" event={"ID":"df48770c-9c54-43d0-87c8-bc7a65bebf2c","Type":"ContainerDied","Data":"6a1e70a0d8fd1d73f15f9f2001698d45c1d4ae8b13a61279c46492aee43ef6de"} Nov 03 22:50:48.558436 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.558193 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9" event={"ID":"df48770c-9c54-43d0-87c8-bc7a65bebf2c","Type":"ContainerDied","Data":"2f67b8640184c2bab8634088622e73311c17e94895d7cbf2a225c91e9b78e9e2"} Nov 03 22:50:48.558436 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.558217 5084 scope.go:122] "RemoveContainer" containerID="6a1e70a0d8fd1d73f15f9f2001698d45c1d4ae8b13a61279c46492aee43ef6de" Nov 03 22:50:48.571352 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.571323 5084 scope.go:122] "RemoveContainer" containerID="6a1e70a0d8fd1d73f15f9f2001698d45c1d4ae8b13a61279c46492aee43ef6de" Nov 03 22:50:48.571740 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:50:48.571716 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a1e70a0d8fd1d73f15f9f2001698d45c1d4ae8b13a61279c46492aee43ef6de\": container with ID starting with 6a1e70a0d8fd1d73f15f9f2001698d45c1d4ae8b13a61279c46492aee43ef6de not found: ID does not exist" containerID="6a1e70a0d8fd1d73f15f9f2001698d45c1d4ae8b13a61279c46492aee43ef6de" Nov 03 22:50:48.571856 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.571747 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a1e70a0d8fd1d73f15f9f2001698d45c1d4ae8b13a61279c46492aee43ef6de"} err="failed to get container status \"6a1e70a0d8fd1d73f15f9f2001698d45c1d4ae8b13a61279c46492aee43ef6de\": rpc error: code = NotFound desc = could not find container \"6a1e70a0d8fd1d73f15f9f2001698d45c1d4ae8b13a61279c46492aee43ef6de\": container with ID starting with 6a1e70a0d8fd1d73f15f9f2001698d45c1d4ae8b13a61279c46492aee43ef6de not found: ID does not exist" Nov 03 22:50:48.727703 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.727668 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9"] Nov 03 22:50:48.731181 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:48.731152 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-345da9/test-model-345da9-kserve-65d869df47-tpjq9"] Nov 03 22:50:50.628963 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:50.628921 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df48770c-9c54-43d0-87c8-bc7a65bebf2c" path="/var/lib/kubelet/pods/df48770c-9c54-43d0-87c8-bc7a65bebf2c/volumes" Nov 03 22:50:56.727473 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:56.727435 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t"] Nov 03 22:50:56.836874 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:56.836829 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh"] Nov 03 22:50:57.456140 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:57.451387 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9"] Nov 03 22:50:57.456140 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:50:57.451683 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" podUID="7e55f64c-dc5c-40c8-ae1f-6b864da55c11" containerName="istio-proxy" containerID="cri-o://01889bd0da0d1984ada1dd7f0451f1347c2d62f6d4c27aed71ae0293c289aa4b" gracePeriod=30 Nov 03 22:51:00.815841 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:00.815785 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw"] Nov 03 22:51:01.728860 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:01.728803 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t" podUID="29c0957d-9d08-4667-b81e-fe1347e4805a" containerName="payload-processing" containerID="cri-o://cb34aea036f918f264d75b7902d7066f5392e39168c88d962663b2a74987722a" gracePeriod=25 Nov 03 22:51:01.838191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:01.838117 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh" podUID="3956759d-6df8-4ae5-b9db-80933d555c9c" containerName="payload-pre-processing" containerID="cri-o://d49d4839593c94fef10dc287faf7aa6219ddfa72964fe8a205e75155c0e27858" gracePeriod=25 Nov 03 22:51:02.039345 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.039313 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t" Nov 03 22:51:02.076102 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.076038 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c0957d-9d08-4667-b81e-fe1347e4805a-kube-api-access-br75g\" (UniqueName: \"kubernetes.io/projected/29c0957d-9d08-4667-b81e-fe1347e4805a-kube-api-access-br75g\") pod \"29c0957d-9d08-4667-b81e-fe1347e4805a\" (UID: \"29c0957d-9d08-4667-b81e-fe1347e4805a\") " Nov 03 22:51:02.076323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.076159 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29c0957d-9d08-4667-b81e-fe1347e4805a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c0957d-9d08-4667-b81e-fe1347e4805a-plugins-config-volume\") pod \"29c0957d-9d08-4667-b81e-fe1347e4805a\" (UID: \"29c0957d-9d08-4667-b81e-fe1347e4805a\") " Nov 03 22:51:02.076632 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.076595 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c0957d-9d08-4667-b81e-fe1347e4805a-plugins-config-volume" pod "29c0957d-9d08-4667-b81e-fe1347e4805a" (UID: "29c0957d-9d08-4667-b81e-fe1347e4805a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:02.076880 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.076853 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c0957d-9d08-4667-b81e-fe1347e4805a-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:02.079210 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.079175 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c0957d-9d08-4667-b81e-fe1347e4805a-kube-api-access-br75g" pod "29c0957d-9d08-4667-b81e-fe1347e4805a" (UID: "29c0957d-9d08-4667-b81e-fe1347e4805a"). InnerVolumeSpecName "kube-api-access-br75g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:02.122968 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.122934 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh" Nov 03 22:51:02.177806 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.177748 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3956759d-6df8-4ae5-b9db-80933d555c9c-kube-api-access-mprf8\" (UniqueName: \"kubernetes.io/projected/3956759d-6df8-4ae5-b9db-80933d555c9c-kube-api-access-mprf8\") pod \"3956759d-6df8-4ae5-b9db-80933d555c9c\" (UID: \"3956759d-6df8-4ae5-b9db-80933d555c9c\") " Nov 03 22:51:02.178010 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.177848 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3956759d-6df8-4ae5-b9db-80933d555c9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3956759d-6df8-4ae5-b9db-80933d555c9c-plugins-config-volume\") pod \"3956759d-6df8-4ae5-b9db-80933d555c9c\" (UID: \"3956759d-6df8-4ae5-b9db-80933d555c9c\") " Nov 03 22:51:02.178301 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.178271 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3956759d-6df8-4ae5-b9db-80933d555c9c-plugins-config-volume" pod "3956759d-6df8-4ae5-b9db-80933d555c9c" (UID: "3956759d-6df8-4ae5-b9db-80933d555c9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:02.178617 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.178598 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br75g\" (UniqueName: \"kubernetes.io/projected/29c0957d-9d08-4667-b81e-fe1347e4805a-kube-api-access-br75g\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:02.178698 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.178619 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3956759d-6df8-4ae5-b9db-80933d555c9c-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:02.180280 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.180245 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3956759d-6df8-4ae5-b9db-80933d555c9c-kube-api-access-mprf8" pod "3956759d-6df8-4ae5-b9db-80933d555c9c" (UID: "3956759d-6df8-4ae5-b9db-80933d555c9c"). InnerVolumeSpecName "kube-api-access-mprf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:02.279664 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.279622 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mprf8\" (UniqueName: \"kubernetes.io/projected/3956759d-6df8-4ae5-b9db-80933d555c9c-kube-api-access-mprf8\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:02.651743 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.651708 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:51:02.679524 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.679487 5084 generic.go:356] "Generic (PLEG): container finished" podID="7e55f64c-dc5c-40c8-ae1f-6b864da55c11" containerID="01889bd0da0d1984ada1dd7f0451f1347c2d62f6d4c27aed71ae0293c289aa4b" exitCode=0 Nov 03 22:51:02.679739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.679545 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" Nov 03 22:51:02.679739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.679572 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" event={"ID":"7e55f64c-dc5c-40c8-ae1f-6b864da55c11","Type":"ContainerDied","Data":"01889bd0da0d1984ada1dd7f0451f1347c2d62f6d4c27aed71ae0293c289aa4b"} Nov 03 22:51:02.679739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.679630 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9" event={"ID":"7e55f64c-dc5c-40c8-ae1f-6b864da55c11","Type":"ContainerDied","Data":"c1ca2b6449d77111ebda306c098830c01e0e84c262be37e22bf311de8ddbb3f9"} Nov 03 22:51:02.679739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.679654 5084 scope.go:122] "RemoveContainer" containerID="01889bd0da0d1984ada1dd7f0451f1347c2d62f6d4c27aed71ae0293c289aa4b" Nov 03 22:51:02.681379 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.681352 5084 generic.go:356] "Generic (PLEG): container finished" podID="3956759d-6df8-4ae5-b9db-80933d555c9c" containerID="d49d4839593c94fef10dc287faf7aa6219ddfa72964fe8a205e75155c0e27858" exitCode=0 Nov 03 22:51:02.681520 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.681421 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh" Nov 03 22:51:02.681520 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.681454 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh" event={"ID":"3956759d-6df8-4ae5-b9db-80933d555c9c","Type":"ContainerDied","Data":"d49d4839593c94fef10dc287faf7aa6219ddfa72964fe8a205e75155c0e27858"} Nov 03 22:51:02.681654 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.681534 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh" event={"ID":"3956759d-6df8-4ae5-b9db-80933d555c9c","Type":"ContainerDied","Data":"f874424d482edf003200b0769de448af25fe48a016bc414f5f094bead71fd967"} Nov 03 22:51:02.683815 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.683782 5084 generic.go:356] "Generic (PLEG): container finished" podID="29c0957d-9d08-4667-b81e-fe1347e4805a" containerID="cb34aea036f918f264d75b7902d7066f5392e39168c88d962663b2a74987722a" exitCode=0 Nov 03 22:51:02.684029 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.683827 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t" event={"ID":"29c0957d-9d08-4667-b81e-fe1347e4805a","Type":"ContainerDied","Data":"cb34aea036f918f264d75b7902d7066f5392e39168c88d962663b2a74987722a"} Nov 03 22:51:02.684134 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.684090 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t" event={"ID":"29c0957d-9d08-4667-b81e-fe1347e4805a","Type":"ContainerDied","Data":"4ccaee9cd8072c3c5bdfa739f5657fcc0f436617259740d19415a1a9629a3839"} Nov 03 22:51:02.684230 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.684022 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t" Nov 03 22:51:02.684557 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.684535 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-podinfo\") pod \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " Nov 03 22:51:02.684646 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.684582 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-kube-api-access-4xnbh\" (UniqueName: \"kubernetes.io/projected/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-kube-api-access-4xnbh\") pod \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " Nov 03 22:51:02.684708 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.684647 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-token\" (UniqueName: \"kubernetes.io/projected/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-token\") pod \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " Nov 03 22:51:02.684766 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.684745 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-workload-socket\") pod \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " Nov 03 22:51:02.684859 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.684828 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-credential-socket\") pod \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " Nov 03 22:51:02.684930 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.684909 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-envoy\") pod \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " Nov 03 22:51:02.684994 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.684958 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-workload-certs\") pod \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " Nov 03 22:51:02.685052 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.685024 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-workload-socket" pod "7e55f64c-dc5c-40c8-ae1f-6b864da55c11" (UID: "7e55f64c-dc5c-40c8-ae1f-6b864da55c11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:02.685161 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.685128 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istiod-ca-cert\") pod \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " Nov 03 22:51:02.685227 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.685208 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-data\") pod \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\" (UID: \"7e55f64c-dc5c-40c8-ae1f-6b864da55c11\") " Nov 03 22:51:02.685227 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.685217 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-credential-socket" pod "7e55f64c-dc5c-40c8-ae1f-6b864da55c11" (UID: "7e55f64c-dc5c-40c8-ae1f-6b864da55c11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:02.685550 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.685405 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-workload-certs" pod "7e55f64c-dc5c-40c8-ae1f-6b864da55c11" (UID: "7e55f64c-dc5c-40c8-ae1f-6b864da55c11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:02.685550 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.685523 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istiod-ca-cert" pod "7e55f64c-dc5c-40c8-ae1f-6b864da55c11" (UID: "7e55f64c-dc5c-40c8-ae1f-6b864da55c11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:02.685762 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.685671 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-data" pod "7e55f64c-dc5c-40c8-ae1f-6b864da55c11" (UID: "7e55f64c-dc5c-40c8-ae1f-6b864da55c11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:02.685961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.685941 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:02.686110 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.685967 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:02.686110 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.685984 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:02.686110 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.685998 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:02.686110 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.686012 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:02.687612 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.687587 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-kube-api-access-4xnbh" pod "7e55f64c-dc5c-40c8-ae1f-6b864da55c11" (UID: "7e55f64c-dc5c-40c8-ae1f-6b864da55c11"). InnerVolumeSpecName "kube-api-access-4xnbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:02.687896 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.687872 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-envoy" pod "7e55f64c-dc5c-40c8-ae1f-6b864da55c11" (UID: "7e55f64c-dc5c-40c8-ae1f-6b864da55c11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:02.688091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.688016 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-podinfo" pod "7e55f64c-dc5c-40c8-ae1f-6b864da55c11" (UID: "7e55f64c-dc5c-40c8-ae1f-6b864da55c11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:02.689115 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.689093 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-token" pod "7e55f64c-dc5c-40c8-ae1f-6b864da55c11" (UID: "7e55f64c-dc5c-40c8-ae1f-6b864da55c11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:02.695582 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.695538 5084 scope.go:122] "RemoveContainer" containerID="01889bd0da0d1984ada1dd7f0451f1347c2d62f6d4c27aed71ae0293c289aa4b" Nov 03 22:51:02.695992 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:51:02.695964 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01889bd0da0d1984ada1dd7f0451f1347c2d62f6d4c27aed71ae0293c289aa4b\": container with ID starting with 01889bd0da0d1984ada1dd7f0451f1347c2d62f6d4c27aed71ae0293c289aa4b not found: ID does not exist" containerID="01889bd0da0d1984ada1dd7f0451f1347c2d62f6d4c27aed71ae0293c289aa4b" Nov 03 22:51:02.696105 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.696007 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01889bd0da0d1984ada1dd7f0451f1347c2d62f6d4c27aed71ae0293c289aa4b"} err="failed to get container status \"01889bd0da0d1984ada1dd7f0451f1347c2d62f6d4c27aed71ae0293c289aa4b\": rpc error: code = NotFound desc = could not find container \"01889bd0da0d1984ada1dd7f0451f1347c2d62f6d4c27aed71ae0293c289aa4b\": container with ID starting with 01889bd0da0d1984ada1dd7f0451f1347c2d62f6d4c27aed71ae0293c289aa4b not found: ID does not exist" Nov 03 22:51:02.696105 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.696051 5084 scope.go:122] "RemoveContainer" containerID="d49d4839593c94fef10dc287faf7aa6219ddfa72964fe8a205e75155c0e27858" Nov 03 22:51:02.707775 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.707743 5084 scope.go:122] "RemoveContainer" containerID="d49d4839593c94fef10dc287faf7aa6219ddfa72964fe8a205e75155c0e27858" Nov 03 22:51:02.708174 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:51:02.708149 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d49d4839593c94fef10dc287faf7aa6219ddfa72964fe8a205e75155c0e27858\": container with ID starting with d49d4839593c94fef10dc287faf7aa6219ddfa72964fe8a205e75155c0e27858 not found: ID does not exist" containerID="d49d4839593c94fef10dc287faf7aa6219ddfa72964fe8a205e75155c0e27858" Nov 03 22:51:02.708260 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.708182 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49d4839593c94fef10dc287faf7aa6219ddfa72964fe8a205e75155c0e27858"} err="failed to get container status \"d49d4839593c94fef10dc287faf7aa6219ddfa72964fe8a205e75155c0e27858\": rpc error: code = NotFound desc = could not find container \"d49d4839593c94fef10dc287faf7aa6219ddfa72964fe8a205e75155c0e27858\": container with ID starting with d49d4839593c94fef10dc287faf7aa6219ddfa72964fe8a205e75155c0e27858 not found: ID does not exist" Nov 03 22:51:02.708260 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.708202 5084 scope.go:122] "RemoveContainer" containerID="cb34aea036f918f264d75b7902d7066f5392e39168c88d962663b2a74987722a" Nov 03 22:51:02.718574 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.718550 5084 scope.go:122] "RemoveContainer" containerID="cb34aea036f918f264d75b7902d7066f5392e39168c88d962663b2a74987722a" Nov 03 22:51:02.718899 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:51:02.718881 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb34aea036f918f264d75b7902d7066f5392e39168c88d962663b2a74987722a\": container with ID starting with cb34aea036f918f264d75b7902d7066f5392e39168c88d962663b2a74987722a not found: ID does not exist" containerID="cb34aea036f918f264d75b7902d7066f5392e39168c88d962663b2a74987722a" Nov 03 22:51:02.718950 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.718909 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb34aea036f918f264d75b7902d7066f5392e39168c88d962663b2a74987722a"} err="failed to get container status \"cb34aea036f918f264d75b7902d7066f5392e39168c88d962663b2a74987722a\": rpc error: code = NotFound desc = could not find container \"cb34aea036f918f264d75b7902d7066f5392e39168c88d962663b2a74987722a\": container with ID starting with cb34aea036f918f264d75b7902d7066f5392e39168c88d962663b2a74987722a not found: ID does not exist" Nov 03 22:51:02.787490 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.787449 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:02.787490 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.787485 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:02.787490 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.787497 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xnbh\" (UniqueName: \"kubernetes.io/projected/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-kube-api-access-4xnbh\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:02.787490 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.787508 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e55f64c-dc5c-40c8-ae1f-6b864da55c11-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:02.888994 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.888899 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh"] Nov 03 22:51:02.894216 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:02.894181 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7bcd0b-9894c8579-q5jjh"] Nov 03 22:51:03.005694 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:03.005657 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t"] Nov 03 22:51:03.009170 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:03.009135 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7bcd0b-7776cb88f6-w6n7t"] Nov 03 22:51:03.181512 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:03.176534 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9"] Nov 03 22:51:03.182082 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:03.182031 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-v2xk9"] Nov 03 22:51:04.629111 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:04.629074 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29c0957d-9d08-4667-b81e-fe1347e4805a" path="/var/lib/kubelet/pods/29c0957d-9d08-4667-b81e-fe1347e4805a/volumes" Nov 03 22:51:04.629540 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:04.629525 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3956759d-6df8-4ae5-b9db-80933d555c9c" path="/var/lib/kubelet/pods/3956759d-6df8-4ae5-b9db-80933d555c9c/volumes" Nov 03 22:51:04.630001 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:04.629985 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e55f64c-dc5c-40c8-ae1f-6b864da55c11" path="/var/lib/kubelet/pods/7e55f64c-dc5c-40c8-ae1f-6b864da55c11/volumes" Nov 03 22:51:05.863378 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:05.863335 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk"] Nov 03 22:51:05.863952 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:05.863640 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" podUID="b94934e9-8f2d-4b49-93f2-a386807c4c79" containerName="maas-api" containerID="cri-o://30b42ec05dd7803d750eec79cfccb119d32173836502f9537412cef3e1e11f32" gracePeriod=30 Nov 03 22:51:05.985905 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:05.985864 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" podUID="b94934e9-8f2d-4b49-93f2-a386807c4c79" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.166:8443/health\": dial tcp 10.132.0.166:8443: connect: connection refused" Nov 03 22:51:06.188676 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.188634 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:51:06.331998 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.331955 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b94934e9-8f2d-4b49-93f2-a386807c4c79-kube-api-access-mk8tf\" (UniqueName: \"kubernetes.io/projected/b94934e9-8f2d-4b49-93f2-a386807c4c79-kube-api-access-mk8tf\") pod \"b94934e9-8f2d-4b49-93f2-a386807c4c79\" (UID: \"b94934e9-8f2d-4b49-93f2-a386807c4c79\") " Nov 03 22:51:06.332209 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.332164 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b94934e9-8f2d-4b49-93f2-a386807c4c79-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94934e9-8f2d-4b49-93f2-a386807c4c79-openshift-service-ca\") pod \"b94934e9-8f2d-4b49-93f2-a386807c4c79\" (UID: \"b94934e9-8f2d-4b49-93f2-a386807c4c79\") " Nov 03 22:51:06.332263 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.332224 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b94934e9-8f2d-4b49-93f2-a386807c4c79-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b94934e9-8f2d-4b49-93f2-a386807c4c79-maas-api-tls\") pod \"b94934e9-8f2d-4b49-93f2-a386807c4c79\" (UID: \"b94934e9-8f2d-4b49-93f2-a386807c4c79\") " Nov 03 22:51:06.332263 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.332258 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b94934e9-8f2d-4b49-93f2-a386807c4c79-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b94934e9-8f2d-4b49-93f2-a386807c4c79-metrics-certs\") pod \"b94934e9-8f2d-4b49-93f2-a386807c4c79\" (UID: \"b94934e9-8f2d-4b49-93f2-a386807c4c79\") " Nov 03 22:51:06.332564 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.332534 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94934e9-8f2d-4b49-93f2-a386807c4c79-openshift-service-ca" pod "b94934e9-8f2d-4b49-93f2-a386807c4c79" (UID: "b94934e9-8f2d-4b49-93f2-a386807c4c79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:06.332867 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.332848 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94934e9-8f2d-4b49-93f2-a386807c4c79-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:06.334963 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.334932 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94934e9-8f2d-4b49-93f2-a386807c4c79-kube-api-access-mk8tf" pod "b94934e9-8f2d-4b49-93f2-a386807c4c79" (UID: "b94934e9-8f2d-4b49-93f2-a386807c4c79"). InnerVolumeSpecName "kube-api-access-mk8tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:06.334963 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.334946 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b94934e9-8f2d-4b49-93f2-a386807c4c79-maas-api-tls" pod "b94934e9-8f2d-4b49-93f2-a386807c4c79" (UID: "b94934e9-8f2d-4b49-93f2-a386807c4c79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:06.335196 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.335176 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b94934e9-8f2d-4b49-93f2-a386807c4c79-metrics-certs" pod "b94934e9-8f2d-4b49-93f2-a386807c4c79" (UID: "b94934e9-8f2d-4b49-93f2-a386807c4c79"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:06.437002 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.436833 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b94934e9-8f2d-4b49-93f2-a386807c4c79-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:06.437002 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.436872 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b94934e9-8f2d-4b49-93f2-a386807c4c79-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:06.437002 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.436890 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk8tf\" (UniqueName: \"kubernetes.io/projected/b94934e9-8f2d-4b49-93f2-a386807c4c79-kube-api-access-mk8tf\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:06.721535 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.721441 5084 generic.go:356] "Generic (PLEG): container finished" podID="b94934e9-8f2d-4b49-93f2-a386807c4c79" containerID="30b42ec05dd7803d750eec79cfccb119d32173836502f9537412cef3e1e11f32" exitCode=0 Nov 03 22:51:06.721535 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.721495 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" Nov 03 22:51:06.721728 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.721528 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" event={"ID":"b94934e9-8f2d-4b49-93f2-a386807c4c79","Type":"ContainerDied","Data":"30b42ec05dd7803d750eec79cfccb119d32173836502f9537412cef3e1e11f32"} Nov 03 22:51:06.721728 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.721590 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk" event={"ID":"b94934e9-8f2d-4b49-93f2-a386807c4c79","Type":"ContainerDied","Data":"c7122e0d44b6588d9a7a5ea4686a4a4a1ee9517bd155b9c6f272a259f37a9191"} Nov 03 22:51:06.721728 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.721620 5084 scope.go:122] "RemoveContainer" containerID="30b42ec05dd7803d750eec79cfccb119d32173836502f9537412cef3e1e11f32" Nov 03 22:51:06.732582 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.732553 5084 scope.go:122] "RemoveContainer" containerID="30b42ec05dd7803d750eec79cfccb119d32173836502f9537412cef3e1e11f32" Nov 03 22:51:06.732969 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:51:06.732944 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30b42ec05dd7803d750eec79cfccb119d32173836502f9537412cef3e1e11f32\": container with ID starting with 30b42ec05dd7803d750eec79cfccb119d32173836502f9537412cef3e1e11f32 not found: ID does not exist" containerID="30b42ec05dd7803d750eec79cfccb119d32173836502f9537412cef3e1e11f32" Nov 03 22:51:06.733108 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.732976 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30b42ec05dd7803d750eec79cfccb119d32173836502f9537412cef3e1e11f32"} err="failed to get container status \"30b42ec05dd7803d750eec79cfccb119d32173836502f9537412cef3e1e11f32\": rpc error: code = NotFound desc = could not find container \"30b42ec05dd7803d750eec79cfccb119d32173836502f9537412cef3e1e11f32\": container with ID starting with 30b42ec05dd7803d750eec79cfccb119d32173836502f9537412cef3e1e11f32 not found: ID does not exist" Nov 03 22:51:06.904887 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.904847 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk"] Nov 03 22:51:06.912842 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:06.912801 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-97965e-658779c4d4-qp9xk"] Nov 03 22:51:08.629544 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:08.629502 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b94934e9-8f2d-4b49-93f2-a386807c4c79" path="/var/lib/kubelet/pods/b94934e9-8f2d-4b49-93f2-a386807c4c79/volumes" Nov 03 22:51:12.025855 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:51:12.025805 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:51:12.862303 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:12.862265 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6"] Nov 03 22:51:12.971570 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:12.971528 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2"] Nov 03 22:51:15.837986 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:15.837919 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" podUID="49089888-27d7-482d-a915-450990214859" containerName="main" containerID="cri-o://030da696d2d7801525333182f4e2a74c1ce59a1074769ea396d1ef88cc939e95" gracePeriod=45 Nov 03 22:51:15.985362 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:15.985309 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" podUID="49089888-27d7-482d-a915-450990214859" containerName="main" probeResult="failure" output="Get \"https://10.132.0.174:8000/ready\": dial tcp 10.132.0.174:8000: connect: connection refused" Nov 03 22:51:16.134856 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.134819 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:51:16.268507 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.268470 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-home\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-home\") pod \"49089888-27d7-482d-a915-450990214859\" (UID: \"49089888-27d7-482d-a915-450990214859\") " Nov 03 22:51:16.268714 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.268555 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-tmp-dir\") pod \"49089888-27d7-482d-a915-450990214859\" (UID: \"49089888-27d7-482d-a915-450990214859\") " Nov 03 22:51:16.268714 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.268671 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49089888-27d7-482d-a915-450990214859-kube-api-access-brkrw\" (UniqueName: \"kubernetes.io/projected/49089888-27d7-482d-a915-450990214859-kube-api-access-brkrw\") pod \"49089888-27d7-482d-a915-450990214859\" (UID: \"49089888-27d7-482d-a915-450990214859\") " Nov 03 22:51:16.268839 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.268715 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-model-cache\") pod \"49089888-27d7-482d-a915-450990214859\" (UID: \"49089888-27d7-482d-a915-450990214859\") " Nov 03 22:51:16.268839 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.268785 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-home" pod "49089888-27d7-482d-a915-450990214859" (UID: "49089888-27d7-482d-a915-450990214859"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:16.268839 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.268823 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-dshm\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-dshm\") pod \"49089888-27d7-482d-a915-450990214859\" (UID: \"49089888-27d7-482d-a915-450990214859\") " Nov 03 22:51:16.268997 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.268888 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-tmp-dir" pod "49089888-27d7-482d-a915-450990214859" (UID: "49089888-27d7-482d-a915-450990214859"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:16.268997 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.268913 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49089888-27d7-482d-a915-450990214859-tls-certs\" (UniqueName: \"kubernetes.io/secret/49089888-27d7-482d-a915-450990214859-tls-certs\") pod \"49089888-27d7-482d-a915-450990214859\" (UID: \"49089888-27d7-482d-a915-450990214859\") " Nov 03 22:51:16.269449 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.269418 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-model-cache" pod "49089888-27d7-482d-a915-450990214859" (UID: "49089888-27d7-482d-a915-450990214859"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:16.270044 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.269484 5084 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-home\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:16.270044 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.269501 5084 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-tmp-dir\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:16.271435 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.271404 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-dshm" pod "49089888-27d7-482d-a915-450990214859" (UID: "49089888-27d7-482d-a915-450990214859"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:16.271662 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.271636 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49089888-27d7-482d-a915-450990214859-kube-api-access-brkrw" pod "49089888-27d7-482d-a915-450990214859" (UID: "49089888-27d7-482d-a915-450990214859"). InnerVolumeSpecName "kube-api-access-brkrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:16.271792 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.271762 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49089888-27d7-482d-a915-450990214859-tls-certs" pod "49089888-27d7-482d-a915-450990214859" (UID: "49089888-27d7-482d-a915-450990214859"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:16.370404 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.370307 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brkrw\" (UniqueName: \"kubernetes.io/projected/49089888-27d7-482d-a915-450990214859-kube-api-access-brkrw\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:16.370404 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.370339 5084 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-model-cache\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:16.370404 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.370352 5084 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49089888-27d7-482d-a915-450990214859-dshm\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:16.370404 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.370365 5084 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49089888-27d7-482d-a915-450990214859-tls-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:16.798572 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.798537 5084 generic.go:356] "Generic (PLEG): container finished" podID="49089888-27d7-482d-a915-450990214859" containerID="030da696d2d7801525333182f4e2a74c1ce59a1074769ea396d1ef88cc939e95" exitCode=0 Nov 03 22:51:16.798756 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.798579 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" Nov 03 22:51:16.798756 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.798605 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" event={"ID":"49089888-27d7-482d-a915-450990214859","Type":"ContainerDied","Data":"030da696d2d7801525333182f4e2a74c1ce59a1074769ea396d1ef88cc939e95"} Nov 03 22:51:16.798756 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.798641 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw" event={"ID":"49089888-27d7-482d-a915-450990214859","Type":"ContainerDied","Data":"08496b308367305fa4a030f5d5c7d652b3a88d8d980206ba5bdbd3b1efef10b9"} Nov 03 22:51:16.798756 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.798660 5084 scope.go:122] "RemoveContainer" containerID="030da696d2d7801525333182f4e2a74c1ce59a1074769ea396d1ef88cc939e95" Nov 03 22:51:16.809182 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.809157 5084 scope.go:122] "RemoveContainer" containerID="030da696d2d7801525333182f4e2a74c1ce59a1074769ea396d1ef88cc939e95" Nov 03 22:51:16.809513 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:51:16.809490 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"030da696d2d7801525333182f4e2a74c1ce59a1074769ea396d1ef88cc939e95\": container with ID starting with 030da696d2d7801525333182f4e2a74c1ce59a1074769ea396d1ef88cc939e95 not found: ID does not exist" containerID="030da696d2d7801525333182f4e2a74c1ce59a1074769ea396d1ef88cc939e95" Nov 03 22:51:16.809573 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.809525 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"030da696d2d7801525333182f4e2a74c1ce59a1074769ea396d1ef88cc939e95"} err="failed to get container status \"030da696d2d7801525333182f4e2a74c1ce59a1074769ea396d1ef88cc939e95\": rpc error: code = NotFound desc = could not find container \"030da696d2d7801525333182f4e2a74c1ce59a1074769ea396d1ef88cc939e95\": container with ID starting with 030da696d2d7801525333182f4e2a74c1ce59a1074769ea396d1ef88cc939e95 not found: ID does not exist" Nov 03 22:51:16.968478 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:16.968441 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw"] Nov 03 22:51:17.018936 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:17.018899 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b30a14/test-model-b30a14-kserve-7f5bcc8d4f-79gzw"] Nov 03 22:51:17.487794 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:17.487754 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6"] Nov 03 22:51:17.488004 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:17.487976 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" podUID="b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2" containerName="maas-api" containerID="cri-o://423be09eb177a2e6b84c416ae20b6f5e1e1087848c81d251740cb50bf6bdd702" gracePeriod=30 Nov 03 22:51:17.820435 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:17.820319 5084 generic.go:356] "Generic (PLEG): container finished" podID="b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2" containerID="423be09eb177a2e6b84c416ae20b6f5e1e1087848c81d251740cb50bf6bdd702" exitCode=0 Nov 03 22:51:17.820618 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:17.820573 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" event={"ID":"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2","Type":"ContainerDied","Data":"423be09eb177a2e6b84c416ae20b6f5e1e1087848c81d251740cb50bf6bdd702"} Nov 03 22:51:17.848388 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:17.848351 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:51:17.863122 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:17.863052 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6" podUID="e9dab1e4-6fcb-49a2-b04d-a803633c232d" containerName="payload-processing" containerID="cri-o://f4e4009578c88f3c18c4bc9dcb2ba798362624cab9e2dc6727e1d81776480a53" gracePeriod=25 Nov 03 22:51:17.927090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:17.923532 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-openshift-service-ca\") pod \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\" (UID: \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\") " Nov 03 22:51:17.927090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:17.923633 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-metrics-certs\") pod \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\" (UID: \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\") " Nov 03 22:51:17.927090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:17.923751 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-kube-api-access-bchkv\" (UniqueName: \"kubernetes.io/projected/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-kube-api-access-bchkv\") pod \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\" (UID: \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\") " Nov 03 22:51:17.927090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:17.923808 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-maas-api-tls\") pod \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\" (UID: \"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2\") " Nov 03 22:51:17.927090 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:17.926859 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-openshift-service-ca" pod "b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2" (UID: "b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:17.941130 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:17.933683 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-kube-api-access-bchkv" pod "b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2" (UID: "b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2"). InnerVolumeSpecName "kube-api-access-bchkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:17.941130 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:17.933781 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-maas-api-tls" pod "b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2" (UID: "b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:17.941130 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:17.941090 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-metrics-certs" pod "b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2" (UID: "b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:17.972038 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:17.971986 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" podUID="9a565b6a-0d39-4a6b-bfeb-af467508b071" containerName="payload-pre-processing" containerID="cri-o://da25510a51ad94bacab8efaf6550928926e4eb81e4fe306050cbf6b31a633922" gracePeriod=25 Nov 03 22:51:18.025139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.025079 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bchkv\" (UniqueName: \"kubernetes.io/projected/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-kube-api-access-bchkv\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:18.025139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.025119 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:18.025139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.025130 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:18.025139 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.025142 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:18.157580 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.157548 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6" Nov 03 22:51:18.227450 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.227401 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9dab1e4-6fcb-49a2-b04d-a803633c232d-kube-api-access-hxwhq\" (UniqueName: \"kubernetes.io/projected/e9dab1e4-6fcb-49a2-b04d-a803633c232d-kube-api-access-hxwhq\") pod \"e9dab1e4-6fcb-49a2-b04d-a803633c232d\" (UID: \"e9dab1e4-6fcb-49a2-b04d-a803633c232d\") " Nov 03 22:51:18.227661 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.227642 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9dab1e4-6fcb-49a2-b04d-a803633c232d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dab1e4-6fcb-49a2-b04d-a803633c232d-plugins-config-volume\") pod \"e9dab1e4-6fcb-49a2-b04d-a803633c232d\" (UID: \"e9dab1e4-6fcb-49a2-b04d-a803633c232d\") " Nov 03 22:51:18.228034 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.228003 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9dab1e4-6fcb-49a2-b04d-a803633c232d-plugins-config-volume" pod "e9dab1e4-6fcb-49a2-b04d-a803633c232d" (UID: "e9dab1e4-6fcb-49a2-b04d-a803633c232d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:18.228194 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.228168 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dab1e4-6fcb-49a2-b04d-a803633c232d-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:18.230053 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.230016 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9dab1e4-6fcb-49a2-b04d-a803633c232d-kube-api-access-hxwhq" pod "e9dab1e4-6fcb-49a2-b04d-a803633c232d" (UID: "e9dab1e4-6fcb-49a2-b04d-a803633c232d"). InnerVolumeSpecName "kube-api-access-hxwhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:18.329782 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.329674 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxwhq\" (UniqueName: \"kubernetes.io/projected/e9dab1e4-6fcb-49a2-b04d-a803633c232d-kube-api-access-hxwhq\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:18.641893 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.641800 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49089888-27d7-482d-a915-450990214859" path="/var/lib/kubelet/pods/49089888-27d7-482d-a915-450990214859/volumes" Nov 03 22:51:18.830912 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.830883 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" Nov 03 22:51:18.830912 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.830895 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" event={"ID":"b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2","Type":"ContainerDied","Data":"05976a1c85c5d85c8f45e45aae7bb106f264cfc344be19d012e716519f569457"} Nov 03 22:51:18.831186 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.830977 5084 scope.go:122] "RemoveContainer" containerID="423be09eb177a2e6b84c416ae20b6f5e1e1087848c81d251740cb50bf6bdd702" Nov 03 22:51:18.832844 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.832791 5084 generic.go:356] "Generic (PLEG): container finished" podID="e9dab1e4-6fcb-49a2-b04d-a803633c232d" containerID="f4e4009578c88f3c18c4bc9dcb2ba798362624cab9e2dc6727e1d81776480a53" exitCode=0 Nov 03 22:51:18.832982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.832856 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6" event={"ID":"e9dab1e4-6fcb-49a2-b04d-a803633c232d","Type":"ContainerDied","Data":"f4e4009578c88f3c18c4bc9dcb2ba798362624cab9e2dc6727e1d81776480a53"} Nov 03 22:51:18.832982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.832918 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6" event={"ID":"e9dab1e4-6fcb-49a2-b04d-a803633c232d","Type":"ContainerDied","Data":"aa2bf90154563b62448cf947e96d91a9ca7fdfa40fcb4f25ead0dec9139671ff"} Nov 03 22:51:18.832982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.832981 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6" Nov 03 22:51:18.842284 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.842257 5084 scope.go:122] "RemoveContainer" containerID="f4e4009578c88f3c18c4bc9dcb2ba798362624cab9e2dc6727e1d81776480a53" Nov 03 22:51:18.855491 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.855458 5084 scope.go:122] "RemoveContainer" containerID="f4e4009578c88f3c18c4bc9dcb2ba798362624cab9e2dc6727e1d81776480a53" Nov 03 22:51:18.856804 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:51:18.856649 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4e4009578c88f3c18c4bc9dcb2ba798362624cab9e2dc6727e1d81776480a53\": container with ID starting with f4e4009578c88f3c18c4bc9dcb2ba798362624cab9e2dc6727e1d81776480a53 not found: ID does not exist" containerID="f4e4009578c88f3c18c4bc9dcb2ba798362624cab9e2dc6727e1d81776480a53" Nov 03 22:51:18.856804 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.856699 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4e4009578c88f3c18c4bc9dcb2ba798362624cab9e2dc6727e1d81776480a53"} err="failed to get container status \"f4e4009578c88f3c18c4bc9dcb2ba798362624cab9e2dc6727e1d81776480a53\": rpc error: code = NotFound desc = could not find container \"f4e4009578c88f3c18c4bc9dcb2ba798362624cab9e2dc6727e1d81776480a53\": container with ID starting with f4e4009578c88f3c18c4bc9dcb2ba798362624cab9e2dc6727e1d81776480a53 not found: ID does not exist" Nov 03 22:51:18.999270 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:18.999123 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6"] Nov 03 22:51:19.002580 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:19.002542 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-345da9-5979d54fcb-4hmq6"] Nov 03 22:51:19.114169 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:19.114135 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6"] Nov 03 22:51:19.121168 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:19.121129 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6"] Nov 03 22:51:20.020291 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:20.020245 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" podUID="9a565b6a-0d39-4a6b-bfeb-af467508b071" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:51:20.628686 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:20.628646 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2" path="/var/lib/kubelet/pods/b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2/volumes" Nov 03 22:51:20.629112 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:20.629096 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9dab1e4-6fcb-49a2-b04d-a803633c232d" path="/var/lib/kubelet/pods/e9dab1e4-6fcb-49a2-b04d-a803633c232d/volumes" Nov 03 22:51:20.802144 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:20.802092 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-345da9-79bffd9848-8xrk6" podUID="b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.153:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:51:30.028487 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:30.028256 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" podUID="9a565b6a-0d39-4a6b-bfeb-af467508b071" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:51:38.251481 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:38.251444 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" Nov 03 22:51:38.301477 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:38.301436 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a565b6a-0d39-4a6b-bfeb-af467508b071-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a565b6a-0d39-4a6b-bfeb-af467508b071-plugins-config-volume\") pod \"9a565b6a-0d39-4a6b-bfeb-af467508b071\" (UID: \"9a565b6a-0d39-4a6b-bfeb-af467508b071\") " Nov 03 22:51:38.301661 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:38.301505 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a565b6a-0d39-4a6b-bfeb-af467508b071-kube-api-access-gkt27\" (UniqueName: \"kubernetes.io/projected/9a565b6a-0d39-4a6b-bfeb-af467508b071-kube-api-access-gkt27\") pod \"9a565b6a-0d39-4a6b-bfeb-af467508b071\" (UID: \"9a565b6a-0d39-4a6b-bfeb-af467508b071\") " Nov 03 22:51:38.301977 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:38.301939 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a565b6a-0d39-4a6b-bfeb-af467508b071-plugins-config-volume" pod "9a565b6a-0d39-4a6b-bfeb-af467508b071" (UID: "9a565b6a-0d39-4a6b-bfeb-af467508b071"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:38.302147 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:38.302110 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a565b6a-0d39-4a6b-bfeb-af467508b071-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:38.304027 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:38.303992 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a565b6a-0d39-4a6b-bfeb-af467508b071-kube-api-access-gkt27" pod "9a565b6a-0d39-4a6b-bfeb-af467508b071" (UID: "9a565b6a-0d39-4a6b-bfeb-af467508b071"). InnerVolumeSpecName "kube-api-access-gkt27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:38.403959 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:38.403898 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkt27\" (UniqueName: \"kubernetes.io/projected/9a565b6a-0d39-4a6b-bfeb-af467508b071-kube-api-access-gkt27\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:38.993120 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:38.993003 5084 generic.go:356] "Generic (PLEG): container finished" podID="9a565b6a-0d39-4a6b-bfeb-af467508b071" containerID="da25510a51ad94bacab8efaf6550928926e4eb81e4fe306050cbf6b31a633922" exitCode=0 Nov 03 22:51:38.993120 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:38.993042 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" Nov 03 22:51:38.993120 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:38.993094 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" event={"ID":"9a565b6a-0d39-4a6b-bfeb-af467508b071","Type":"ContainerDied","Data":"da25510a51ad94bacab8efaf6550928926e4eb81e4fe306050cbf6b31a633922"} Nov 03 22:51:38.993389 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:38.993155 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2" event={"ID":"9a565b6a-0d39-4a6b-bfeb-af467508b071","Type":"ContainerDied","Data":"e0ec274743f7bdbd8144818372ee08869951401d8dd7fa342503a0ed9040d919"} Nov 03 22:51:38.993389 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:38.993184 5084 scope.go:122] "RemoveContainer" containerID="da25510a51ad94bacab8efaf6550928926e4eb81e4fe306050cbf6b31a633922" Nov 03 22:51:39.004045 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:39.004013 5084 scope.go:122] "RemoveContainer" containerID="da25510a51ad94bacab8efaf6550928926e4eb81e4fe306050cbf6b31a633922" Nov 03 22:51:39.004466 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:51:39.004440 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da25510a51ad94bacab8efaf6550928926e4eb81e4fe306050cbf6b31a633922\": container with ID starting with da25510a51ad94bacab8efaf6550928926e4eb81e4fe306050cbf6b31a633922 not found: ID does not exist" containerID="da25510a51ad94bacab8efaf6550928926e4eb81e4fe306050cbf6b31a633922" Nov 03 22:51:39.004583 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:39.004473 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da25510a51ad94bacab8efaf6550928926e4eb81e4fe306050cbf6b31a633922"} err="failed to get container status \"da25510a51ad94bacab8efaf6550928926e4eb81e4fe306050cbf6b31a633922\": rpc error: code = NotFound desc = could not find container \"da25510a51ad94bacab8efaf6550928926e4eb81e4fe306050cbf6b31a633922\": container with ID starting with da25510a51ad94bacab8efaf6550928926e4eb81e4fe306050cbf6b31a633922 not found: ID does not exist" Nov 03 22:51:39.160460 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:39.160409 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2"] Nov 03 22:51:39.211847 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:39.211804 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-345da9-85cbc4fd5f-cfjw2"] Nov 03 22:51:39.465397 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:39.465355 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b"] Nov 03 22:51:39.701532 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:39.701497 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7"] Nov 03 22:51:39.701750 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:39.701727 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" podUID="429389e2-f479-4c31-9170-bf11ff231d48" containerName="maas-api" containerID="cri-o://9726ea99f773993eedec23e63909f6fef54878b04ee5ef05bda9f46a8d8bb7e2" gracePeriod=30 Nov 03 22:51:40.002687 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:40.002650 5084 generic.go:356] "Generic (PLEG): container finished" podID="429389e2-f479-4c31-9170-bf11ff231d48" containerID="9726ea99f773993eedec23e63909f6fef54878b04ee5ef05bda9f46a8d8bb7e2" exitCode=0 Nov 03 22:51:40.002905 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:40.002726 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" event={"ID":"429389e2-f479-4c31-9170-bf11ff231d48","Type":"ContainerDied","Data":"9726ea99f773993eedec23e63909f6fef54878b04ee5ef05bda9f46a8d8bb7e2"} Nov 03 22:51:40.002905 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:40.002787 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" event={"ID":"429389e2-f479-4c31-9170-bf11ff231d48","Type":"ContainerDied","Data":"0f227412d8b773dd9a7ab3335662df4859fb79ef6c38e2d8f3825226990b6e64"} Nov 03 22:51:40.002905 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:40.002815 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f227412d8b773dd9a7ab3335662df4859fb79ef6c38e2d8f3825226990b6e64" Nov 03 22:51:40.016801 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:40.016773 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:51:40.125686 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:40.125543 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/429389e2-f479-4c31-9170-bf11ff231d48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/429389e2-f479-4c31-9170-bf11ff231d48-openshift-service-ca\") pod \"429389e2-f479-4c31-9170-bf11ff231d48\" (UID: \"429389e2-f479-4c31-9170-bf11ff231d48\") " Nov 03 22:51:40.125686 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:40.125614 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/429389e2-f479-4c31-9170-bf11ff231d48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/429389e2-f479-4c31-9170-bf11ff231d48-maas-api-tls\") pod \"429389e2-f479-4c31-9170-bf11ff231d48\" (UID: \"429389e2-f479-4c31-9170-bf11ff231d48\") " Nov 03 22:51:40.125975 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:40.125697 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/429389e2-f479-4c31-9170-bf11ff231d48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/429389e2-f479-4c31-9170-bf11ff231d48-metrics-certs\") pod \"429389e2-f479-4c31-9170-bf11ff231d48\" (UID: \"429389e2-f479-4c31-9170-bf11ff231d48\") " Nov 03 22:51:40.125975 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:40.125730 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/429389e2-f479-4c31-9170-bf11ff231d48-kube-api-access-fbx6w\" (UniqueName: \"kubernetes.io/projected/429389e2-f479-4c31-9170-bf11ff231d48-kube-api-access-fbx6w\") pod \"429389e2-f479-4c31-9170-bf11ff231d48\" (UID: \"429389e2-f479-4c31-9170-bf11ff231d48\") " Nov 03 22:51:40.127957 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:40.127924 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429389e2-f479-4c31-9170-bf11ff231d48-openshift-service-ca" pod "429389e2-f479-4c31-9170-bf11ff231d48" (UID: "429389e2-f479-4c31-9170-bf11ff231d48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:40.129380 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:40.129255 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/429389e2-f479-4c31-9170-bf11ff231d48-maas-api-tls" pod "429389e2-f479-4c31-9170-bf11ff231d48" (UID: "429389e2-f479-4c31-9170-bf11ff231d48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:40.129380 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:40.129348 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/429389e2-f479-4c31-9170-bf11ff231d48-metrics-certs" pod "429389e2-f479-4c31-9170-bf11ff231d48" (UID: "429389e2-f479-4c31-9170-bf11ff231d48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:40.130277 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:40.130253 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429389e2-f479-4c31-9170-bf11ff231d48-kube-api-access-fbx6w" pod "429389e2-f479-4c31-9170-bf11ff231d48" (UID: "429389e2-f479-4c31-9170-bf11ff231d48"). InnerVolumeSpecName "kube-api-access-fbx6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:40.227296 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:40.227257 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/429389e2-f479-4c31-9170-bf11ff231d48-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:40.227296 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:40.227287 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/429389e2-f479-4c31-9170-bf11ff231d48-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:40.227296 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:40.227297 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/429389e2-f479-4c31-9170-bf11ff231d48-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:40.227553 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:40.227306 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbx6w\" (UniqueName: \"kubernetes.io/projected/429389e2-f479-4c31-9170-bf11ff231d48-kube-api-access-fbx6w\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:40.644695 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:40.644556 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a565b6a-0d39-4a6b-bfeb-af467508b071" path="/var/lib/kubelet/pods/9a565b6a-0d39-4a6b-bfeb-af467508b071/volumes" Nov 03 22:51:41.010370 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:41.010282 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7" Nov 03 22:51:41.186875 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:41.186829 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7"] Nov 03 22:51:41.236212 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:41.236162 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7bcd0b-854d4b77d6-w5wn7"] Nov 03 22:51:42.023831 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:51:42.023793 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:51:42.628134 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:42.628091 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="429389e2-f479-4c31-9170-bf11ff231d48" path="/var/lib/kubelet/pods/429389e2-f479-4c31-9170-bf11ff231d48/volumes" Nov 03 22:51:44.467215 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:44.467165 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" podUID="be288892-c462-4214-a907-c7a2b7b75c25" containerName="payload-pre-processing" containerID="cri-o://6b4d813871421b658f3bb1f744219de8c5d4e168681a2891188c1a3e1638264a" gracePeriod=25 Nov 03 22:51:48.599130 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:48.599040 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" podUID="be288892-c462-4214-a907-c7a2b7b75c25" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:51:51.111544 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:51.111501 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz"] Nov 03 22:51:51.112026 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:51.111739 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" podUID="f2ea5507-0676-4e91-a1fd-b13b524cd473" containerName="maas-api" containerID="cri-o://e9b3fcc0ec2f85f254c12e9dc5cb8676bf3cdedcd12b3e8997137e9c3f2a389c" gracePeriod=30 Nov 03 22:51:51.411194 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:51.411163 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:51:51.581823 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:51.581776 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ea5507-0676-4e91-a1fd-b13b524cd473-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2ea5507-0676-4e91-a1fd-b13b524cd473-maas-api-tls\") pod \"f2ea5507-0676-4e91-a1fd-b13b524cd473\" (UID: \"f2ea5507-0676-4e91-a1fd-b13b524cd473\") " Nov 03 22:51:51.582021 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:51.581904 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ea5507-0676-4e91-a1fd-b13b524cd473-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ea5507-0676-4e91-a1fd-b13b524cd473-openshift-service-ca\") pod \"f2ea5507-0676-4e91-a1fd-b13b524cd473\" (UID: \"f2ea5507-0676-4e91-a1fd-b13b524cd473\") " Nov 03 22:51:51.582021 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:51.581942 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ea5507-0676-4e91-a1fd-b13b524cd473-kube-api-access-9pvt4\" (UniqueName: \"kubernetes.io/projected/f2ea5507-0676-4e91-a1fd-b13b524cd473-kube-api-access-9pvt4\") pod \"f2ea5507-0676-4e91-a1fd-b13b524cd473\" (UID: \"f2ea5507-0676-4e91-a1fd-b13b524cd473\") " Nov 03 22:51:51.582021 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:51.581985 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ea5507-0676-4e91-a1fd-b13b524cd473-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2ea5507-0676-4e91-a1fd-b13b524cd473-metrics-certs\") pod \"f2ea5507-0676-4e91-a1fd-b13b524cd473\" (UID: \"f2ea5507-0676-4e91-a1fd-b13b524cd473\") " Nov 03 22:51:51.582417 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:51.582387 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ea5507-0676-4e91-a1fd-b13b524cd473-openshift-service-ca" pod "f2ea5507-0676-4e91-a1fd-b13b524cd473" (UID: "f2ea5507-0676-4e91-a1fd-b13b524cd473"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:51.582687 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:51.582667 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ea5507-0676-4e91-a1fd-b13b524cd473-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:51.584764 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:51.584728 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ea5507-0676-4e91-a1fd-b13b524cd473-kube-api-access-9pvt4" pod "f2ea5507-0676-4e91-a1fd-b13b524cd473" (UID: "f2ea5507-0676-4e91-a1fd-b13b524cd473"). InnerVolumeSpecName "kube-api-access-9pvt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:51.584764 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:51.584747 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ea5507-0676-4e91-a1fd-b13b524cd473-maas-api-tls" pod "f2ea5507-0676-4e91-a1fd-b13b524cd473" (UID: "f2ea5507-0676-4e91-a1fd-b13b524cd473"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:51.584946 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:51.584759 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ea5507-0676-4e91-a1fd-b13b524cd473-metrics-certs" pod "f2ea5507-0676-4e91-a1fd-b13b524cd473" (UID: "f2ea5507-0676-4e91-a1fd-b13b524cd473"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:51.684113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:51.684004 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2ea5507-0676-4e91-a1fd-b13b524cd473-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:51.684113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:51.684042 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pvt4\" (UniqueName: \"kubernetes.io/projected/f2ea5507-0676-4e91-a1fd-b13b524cd473-kube-api-access-9pvt4\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:51.684113 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:51.684055 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2ea5507-0676-4e91-a1fd-b13b524cd473-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:52.085602 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:52.085568 5084 generic.go:356] "Generic (PLEG): container finished" podID="f2ea5507-0676-4e91-a1fd-b13b524cd473" containerID="e9b3fcc0ec2f85f254c12e9dc5cb8676bf3cdedcd12b3e8997137e9c3f2a389c" exitCode=0 Nov 03 22:51:52.085815 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:52.085618 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" Nov 03 22:51:52.085815 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:52.085652 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" event={"ID":"f2ea5507-0676-4e91-a1fd-b13b524cd473","Type":"ContainerDied","Data":"e9b3fcc0ec2f85f254c12e9dc5cb8676bf3cdedcd12b3e8997137e9c3f2a389c"} Nov 03 22:51:52.085815 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:52.085712 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz" event={"ID":"f2ea5507-0676-4e91-a1fd-b13b524cd473","Type":"ContainerDied","Data":"a8c6905760c5978a51a921f0adf89c3f2d1d6e2174e35978da6fe042d59a2fc2"} Nov 03 22:51:52.085815 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:52.085736 5084 scope.go:122] "RemoveContainer" containerID="e9b3fcc0ec2f85f254c12e9dc5cb8676bf3cdedcd12b3e8997137e9c3f2a389c" Nov 03 22:51:52.097997 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:52.097966 5084 scope.go:122] "RemoveContainer" containerID="e9b3fcc0ec2f85f254c12e9dc5cb8676bf3cdedcd12b3e8997137e9c3f2a389c" Nov 03 22:51:52.098388 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:51:52.098366 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b3fcc0ec2f85f254c12e9dc5cb8676bf3cdedcd12b3e8997137e9c3f2a389c\": container with ID starting with e9b3fcc0ec2f85f254c12e9dc5cb8676bf3cdedcd12b3e8997137e9c3f2a389c not found: ID does not exist" containerID="e9b3fcc0ec2f85f254c12e9dc5cb8676bf3cdedcd12b3e8997137e9c3f2a389c" Nov 03 22:51:52.098462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:52.098399 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b3fcc0ec2f85f254c12e9dc5cb8676bf3cdedcd12b3e8997137e9c3f2a389c"} err="failed to get container status \"e9b3fcc0ec2f85f254c12e9dc5cb8676bf3cdedcd12b3e8997137e9c3f2a389c\": rpc error: code = NotFound desc = could not find container \"e9b3fcc0ec2f85f254c12e9dc5cb8676bf3cdedcd12b3e8997137e9c3f2a389c\": container with ID starting with e9b3fcc0ec2f85f254c12e9dc5cb8676bf3cdedcd12b3e8997137e9c3f2a389c not found: ID does not exist" Nov 03 22:51:52.269574 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:52.269536 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz"] Nov 03 22:51:52.324211 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:52.324161 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-295682-848544cd6b-2g5cz"] Nov 03 22:51:52.628716 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:52.628672 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ea5507-0676-4e91-a1fd-b13b524cd473" path="/var/lib/kubelet/pods/f2ea5507-0676-4e91-a1fd-b13b524cd473/volumes" Nov 03 22:51:54.072287 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:54.072247 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7"] Nov 03 22:51:57.521233 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:57.521188 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4"] Nov 03 22:51:57.521740 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:57.521543 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" podUID="4e4d71d8-dfc0-437a-a639-8e763b69f249" containerName="maas-api" containerID="cri-o://3028118b05fb3004bd2e0b373180a9c5805fe763b863052b4bc695774559eef9" gracePeriod=30 Nov 03 22:51:57.848519 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:57.848481 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:51:57.859182 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:57.859147 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4d71d8-dfc0-437a-a639-8e763b69f249-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4d71d8-dfc0-437a-a639-8e763b69f249-openshift-service-ca\") pod \"4e4d71d8-dfc0-437a-a639-8e763b69f249\" (UID: \"4e4d71d8-dfc0-437a-a639-8e763b69f249\") " Nov 03 22:51:57.859390 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:57.859232 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e4d71d8-dfc0-437a-a639-8e763b69f249-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4d71d8-dfc0-437a-a639-8e763b69f249-metrics-certs\") pod \"4e4d71d8-dfc0-437a-a639-8e763b69f249\" (UID: \"4e4d71d8-dfc0-437a-a639-8e763b69f249\") " Nov 03 22:51:57.859390 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:57.859270 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4d71d8-dfc0-437a-a639-8e763b69f249-kube-api-access-mrngq\" (UniqueName: \"kubernetes.io/projected/4e4d71d8-dfc0-437a-a639-8e763b69f249-kube-api-access-mrngq\") pod \"4e4d71d8-dfc0-437a-a639-8e763b69f249\" (UID: \"4e4d71d8-dfc0-437a-a639-8e763b69f249\") " Nov 03 22:51:57.859390 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:57.859386 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e4d71d8-dfc0-437a-a639-8e763b69f249-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4d71d8-dfc0-437a-a639-8e763b69f249-maas-api-tls\") pod \"4e4d71d8-dfc0-437a-a639-8e763b69f249\" (UID: \"4e4d71d8-dfc0-437a-a639-8e763b69f249\") " Nov 03 22:51:57.859706 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:57.859661 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4d71d8-dfc0-437a-a639-8e763b69f249-openshift-service-ca" pod "4e4d71d8-dfc0-437a-a639-8e763b69f249" (UID: "4e4d71d8-dfc0-437a-a639-8e763b69f249"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:57.859858 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:57.859767 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4d71d8-dfc0-437a-a639-8e763b69f249-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:57.861689 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:57.861651 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e4d71d8-dfc0-437a-a639-8e763b69f249-metrics-certs" pod "4e4d71d8-dfc0-437a-a639-8e763b69f249" (UID: "4e4d71d8-dfc0-437a-a639-8e763b69f249"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:57.861862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:57.861836 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e4d71d8-dfc0-437a-a639-8e763b69f249-maas-api-tls" pod "4e4d71d8-dfc0-437a-a639-8e763b69f249" (UID: "4e4d71d8-dfc0-437a-a639-8e763b69f249"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:57.862521 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:57.862493 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4d71d8-dfc0-437a-a639-8e763b69f249-kube-api-access-mrngq" pod "4e4d71d8-dfc0-437a-a639-8e763b69f249" (UID: "4e4d71d8-dfc0-437a-a639-8e763b69f249"). InnerVolumeSpecName "kube-api-access-mrngq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:57.981119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:57.977562 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4d71d8-dfc0-437a-a639-8e763b69f249-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:57.981119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:57.977602 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrngq\" (UniqueName: \"kubernetes.io/projected/4e4d71d8-dfc0-437a-a639-8e763b69f249-kube-api-access-mrngq\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:57.981119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:57.977619 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4d71d8-dfc0-437a-a639-8e763b69f249-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:58.140756 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:58.140659 5084 generic.go:356] "Generic (PLEG): container finished" podID="4e4d71d8-dfc0-437a-a639-8e763b69f249" containerID="3028118b05fb3004bd2e0b373180a9c5805fe763b863052b4bc695774559eef9" exitCode=0 Nov 03 22:51:58.140756 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:58.140725 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" Nov 03 22:51:58.140965 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:58.140745 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" event={"ID":"4e4d71d8-dfc0-437a-a639-8e763b69f249","Type":"ContainerDied","Data":"3028118b05fb3004bd2e0b373180a9c5805fe763b863052b4bc695774559eef9"} Nov 03 22:51:58.140965 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:58.140837 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4" event={"ID":"4e4d71d8-dfc0-437a-a639-8e763b69f249","Type":"ContainerDied","Data":"3ebccfe542d977c9fc81512de387babde9ee6a64858c3256d48c506f0e4cec92"} Nov 03 22:51:58.140965 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:58.140861 5084 scope.go:122] "RemoveContainer" containerID="3028118b05fb3004bd2e0b373180a9c5805fe763b863052b4bc695774559eef9" Nov 03 22:51:58.152355 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:58.152327 5084 scope.go:122] "RemoveContainer" containerID="3028118b05fb3004bd2e0b373180a9c5805fe763b863052b4bc695774559eef9" Nov 03 22:51:58.152715 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:51:58.152692 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3028118b05fb3004bd2e0b373180a9c5805fe763b863052b4bc695774559eef9\": container with ID starting with 3028118b05fb3004bd2e0b373180a9c5805fe763b863052b4bc695774559eef9 not found: ID does not exist" containerID="3028118b05fb3004bd2e0b373180a9c5805fe763b863052b4bc695774559eef9" Nov 03 22:51:58.152768 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:58.152728 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3028118b05fb3004bd2e0b373180a9c5805fe763b863052b4bc695774559eef9"} err="failed to get container status \"3028118b05fb3004bd2e0b373180a9c5805fe763b863052b4bc695774559eef9\": rpc error: code = NotFound desc = could not find container \"3028118b05fb3004bd2e0b373180a9c5805fe763b863052b4bc695774559eef9\": container with ID starting with 3028118b05fb3004bd2e0b373180a9c5805fe763b863052b4bc695774559eef9 not found: ID does not exist" Nov 03 22:51:58.311509 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:58.311468 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4"] Nov 03 22:51:58.315834 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:58.315797 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f61714-7dd9f5fc6d-ctsx4"] Nov 03 22:51:58.600755 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:58.600701 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" podUID="be288892-c462-4214-a907-c7a2b7b75c25" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:51:58.628684 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:58.628646 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4d71d8-dfc0-437a-a639-8e763b69f249" path="/var/lib/kubelet/pods/4e4d71d8-dfc0-437a-a639-8e763b69f249/volumes" Nov 03 22:51:58.850327 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:58.849116 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp"] Nov 03 22:51:58.850327 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:58.849377 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" podUID="912af5f3-b029-4368-b74b-693937c9431d" containerName="maas-api" containerID="cri-o://4fe9b37e4c339022f8d2c9f323878cd25f6f6ecfcccdff749318d48ebe12d92e" gracePeriod=30 Nov 03 22:51:59.150415 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:59.150377 5084 generic.go:356] "Generic (PLEG): container finished" podID="912af5f3-b029-4368-b74b-693937c9431d" containerID="4fe9b37e4c339022f8d2c9f323878cd25f6f6ecfcccdff749318d48ebe12d92e" exitCode=0 Nov 03 22:51:59.150589 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:59.150445 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" event={"ID":"912af5f3-b029-4368-b74b-693937c9431d","Type":"ContainerDied","Data":"4fe9b37e4c339022f8d2c9f323878cd25f6f6ecfcccdff749318d48ebe12d92e"} Nov 03 22:51:59.150589 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:59.150508 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" event={"ID":"912af5f3-b029-4368-b74b-693937c9431d","Type":"ContainerDied","Data":"e56d3d539eca0201b191c2b93cd11c7a0c010eb4172cda9004ff159772be8289"} Nov 03 22:51:59.150589 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:59.150530 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e56d3d539eca0201b191c2b93cd11c7a0c010eb4172cda9004ff159772be8289" Nov 03 22:51:59.165944 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:59.165911 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:51:59.197790 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:59.197742 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/912af5f3-b029-4368-b74b-693937c9431d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/912af5f3-b029-4368-b74b-693937c9431d-metrics-certs\") pod \"912af5f3-b029-4368-b74b-693937c9431d\" (UID: \"912af5f3-b029-4368-b74b-693937c9431d\") " Nov 03 22:51:59.198002 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:59.197800 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/912af5f3-b029-4368-b74b-693937c9431d-kube-api-access-tx2gh\" (UniqueName: \"kubernetes.io/projected/912af5f3-b029-4368-b74b-693937c9431d-kube-api-access-tx2gh\") pod \"912af5f3-b029-4368-b74b-693937c9431d\" (UID: \"912af5f3-b029-4368-b74b-693937c9431d\") " Nov 03 22:51:59.198002 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:59.197846 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/912af5f3-b029-4368-b74b-693937c9431d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/912af5f3-b029-4368-b74b-693937c9431d-openshift-service-ca\") pod \"912af5f3-b029-4368-b74b-693937c9431d\" (UID: \"912af5f3-b029-4368-b74b-693937c9431d\") " Nov 03 22:51:59.198002 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:59.197944 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/912af5f3-b029-4368-b74b-693937c9431d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/912af5f3-b029-4368-b74b-693937c9431d-maas-api-tls\") pod \"912af5f3-b029-4368-b74b-693937c9431d\" (UID: \"912af5f3-b029-4368-b74b-693937c9431d\") " Nov 03 22:51:59.198268 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:59.198242 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/912af5f3-b029-4368-b74b-693937c9431d-openshift-service-ca" pod "912af5f3-b029-4368-b74b-693937c9431d" (UID: "912af5f3-b029-4368-b74b-693937c9431d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:59.198581 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:59.198556 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/912af5f3-b029-4368-b74b-693937c9431d-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:59.200683 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:59.200644 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/912af5f3-b029-4368-b74b-693937c9431d-maas-api-tls" pod "912af5f3-b029-4368-b74b-693937c9431d" (UID: "912af5f3-b029-4368-b74b-693937c9431d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:59.200952 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:59.200932 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912af5f3-b029-4368-b74b-693937c9431d-kube-api-access-tx2gh" pod "912af5f3-b029-4368-b74b-693937c9431d" (UID: "912af5f3-b029-4368-b74b-693937c9431d"). InnerVolumeSpecName "kube-api-access-tx2gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:59.201055 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:59.201039 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/912af5f3-b029-4368-b74b-693937c9431d-metrics-certs" pod "912af5f3-b029-4368-b74b-693937c9431d" (UID: "912af5f3-b029-4368-b74b-693937c9431d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:59.299927 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:59.299881 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/912af5f3-b029-4368-b74b-693937c9431d-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:59.300171 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:59.299976 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/912af5f3-b029-4368-b74b-693937c9431d-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:51:59.300171 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:51:59.299997 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx2gh\" (UniqueName: \"kubernetes.io/projected/912af5f3-b029-4368-b74b-693937c9431d-kube-api-access-tx2gh\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:00.166568 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:00.166508 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp" Nov 03 22:52:00.341154 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:00.341110 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp"] Nov 03 22:52:00.390159 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:00.390120 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-472b46-7b79c6c9fc-pk7vp"] Nov 03 22:52:00.628319 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:00.628281 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="912af5f3-b029-4368-b74b-693937c9431d" path="/var/lib/kubelet/pods/912af5f3-b029-4368-b74b-693937c9431d/volumes" Nov 03 22:52:04.751634 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:04.751597 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" Nov 03 22:52:04.785719 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:04.785674 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be288892-c462-4214-a907-c7a2b7b75c25-kube-api-access-lrmkw\" (UniqueName: \"kubernetes.io/projected/be288892-c462-4214-a907-c7a2b7b75c25-kube-api-access-lrmkw\") pod \"be288892-c462-4214-a907-c7a2b7b75c25\" (UID: \"be288892-c462-4214-a907-c7a2b7b75c25\") " Nov 03 22:52:04.785949 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:04.785829 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be288892-c462-4214-a907-c7a2b7b75c25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be288892-c462-4214-a907-c7a2b7b75c25-plugins-config-volume\") pod \"be288892-c462-4214-a907-c7a2b7b75c25\" (UID: \"be288892-c462-4214-a907-c7a2b7b75c25\") " Nov 03 22:52:04.786284 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:04.786258 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be288892-c462-4214-a907-c7a2b7b75c25-plugins-config-volume" pod "be288892-c462-4214-a907-c7a2b7b75c25" (UID: "be288892-c462-4214-a907-c7a2b7b75c25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:04.788000 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:04.787969 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be288892-c462-4214-a907-c7a2b7b75c25-kube-api-access-lrmkw" pod "be288892-c462-4214-a907-c7a2b7b75c25" (UID: "be288892-c462-4214-a907-c7a2b7b75c25"). InnerVolumeSpecName "kube-api-access-lrmkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:04.887792 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:04.887743 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be288892-c462-4214-a907-c7a2b7b75c25-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:04.887792 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:04.887793 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrmkw\" (UniqueName: \"kubernetes.io/projected/be288892-c462-4214-a907-c7a2b7b75c25-kube-api-access-lrmkw\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:05.206925 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:05.206889 5084 generic.go:356] "Generic (PLEG): container finished" podID="be288892-c462-4214-a907-c7a2b7b75c25" containerID="6b4d813871421b658f3bb1f744219de8c5d4e168681a2891188c1a3e1638264a" exitCode=0 Nov 03 22:52:05.207158 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:05.206934 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" event={"ID":"be288892-c462-4214-a907-c7a2b7b75c25","Type":"ContainerDied","Data":"6b4d813871421b658f3bb1f744219de8c5d4e168681a2891188c1a3e1638264a"} Nov 03 22:52:05.207158 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:05.206944 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" Nov 03 22:52:05.207158 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:05.206988 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b" event={"ID":"be288892-c462-4214-a907-c7a2b7b75c25","Type":"ContainerDied","Data":"bd34da136fb5cc81ece59ac785ad3e35ed16b69a6e0b6bea832af454b74c027f"} Nov 03 22:52:05.207158 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:05.207007 5084 scope.go:122] "RemoveContainer" containerID="6b4d813871421b658f3bb1f744219de8c5d4e168681a2891188c1a3e1638264a" Nov 03 22:52:05.220447 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:05.220418 5084 scope.go:122] "RemoveContainer" containerID="6b4d813871421b658f3bb1f744219de8c5d4e168681a2891188c1a3e1638264a" Nov 03 22:52:05.220846 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:52:05.220815 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b4d813871421b658f3bb1f744219de8c5d4e168681a2891188c1a3e1638264a\": container with ID starting with 6b4d813871421b658f3bb1f744219de8c5d4e168681a2891188c1a3e1638264a not found: ID does not exist" containerID="6b4d813871421b658f3bb1f744219de8c5d4e168681a2891188c1a3e1638264a" Nov 03 22:52:05.220979 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:05.220855 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b4d813871421b658f3bb1f744219de8c5d4e168681a2891188c1a3e1638264a"} err="failed to get container status \"6b4d813871421b658f3bb1f744219de8c5d4e168681a2891188c1a3e1638264a\": rpc error: code = NotFound desc = could not find container \"6b4d813871421b658f3bb1f744219de8c5d4e168681a2891188c1a3e1638264a\": container with ID starting with 6b4d813871421b658f3bb1f744219de8c5d4e168681a2891188c1a3e1638264a not found: ID does not exist" Nov 03 22:52:05.374372 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:05.374331 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b"] Nov 03 22:52:05.381739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:05.381696 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b30a14-5b6864cdb7-vkn4b"] Nov 03 22:52:06.630124 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:06.630042 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be288892-c462-4214-a907-c7a2b7b75c25" path="/var/lib/kubelet/pods/be288892-c462-4214-a907-c7a2b7b75c25/volumes" Nov 03 22:52:06.773458 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:06.773414 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh"] Nov 03 22:52:06.773717 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:06.773687 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" podUID="aa2863b4-3169-423e-a0ee-65fd49117624" containerName="istio-proxy" containerID="cri-o://60f124f6b353f0ae2aa2400e506f6441d968f2dfea4df98c3f0e425bae44889b" gracePeriod=30 Nov 03 22:52:09.094000 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.093954 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" containerID="cri-o://29e4a30937d211502610c2ac9b6d52353c9c9fdd318c5c9ab672944c09770478" gracePeriod=45 Nov 03 22:52:09.237301 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.237085 5084 generic.go:356] "Generic (PLEG): container finished" podID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerID="29e4a30937d211502610c2ac9b6d52353c9c9fdd318c5c9ab672944c09770478" exitCode=0 Nov 03 22:52:09.237301 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.237104 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" event={"ID":"8b0fcd09-e958-4cbb-9881-c6e6488cb39d","Type":"ContainerDied","Data":"29e4a30937d211502610c2ac9b6d52353c9c9fdd318c5c9ab672944c09770478"} Nov 03 22:52:09.376343 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.376304 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:52:09.443901 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.443857 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-tls-certs\") pod \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " Nov 03 22:52:09.444158 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.443921 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-tmp-dir\") pod \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " Nov 03 22:52:09.444158 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.443944 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-dshm\") pod \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " Nov 03 22:52:09.444158 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.443982 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-kube-api-access-kc6w6\" (UniqueName: \"kubernetes.io/projected/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-kube-api-access-kc6w6\") pod \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " Nov 03 22:52:09.444158 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.444008 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-model-cache\") pod \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " Nov 03 22:52:09.444158 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.444028 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-home\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-home\") pod \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\" (UID: \"8b0fcd09-e958-4cbb-9881-c6e6488cb39d\") " Nov 03 22:52:09.444456 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.444298 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-tmp-dir" pod "8b0fcd09-e958-4cbb-9881-c6e6488cb39d" (UID: "8b0fcd09-e958-4cbb-9881-c6e6488cb39d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:09.444456 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.444361 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-model-cache" pod "8b0fcd09-e958-4cbb-9881-c6e6488cb39d" (UID: "8b0fcd09-e958-4cbb-9881-c6e6488cb39d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:09.444567 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.444478 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-home" pod "8b0fcd09-e958-4cbb-9881-c6e6488cb39d" (UID: "8b0fcd09-e958-4cbb-9881-c6e6488cb39d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:09.444623 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.444611 5084 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-tmp-dir\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.444673 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.444633 5084 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-model-cache\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.446479 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.446452 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-dshm" pod "8b0fcd09-e958-4cbb-9881-c6e6488cb39d" (UID: "8b0fcd09-e958-4cbb-9881-c6e6488cb39d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:09.446479 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.446474 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-kube-api-access-kc6w6" pod "8b0fcd09-e958-4cbb-9881-c6e6488cb39d" (UID: "8b0fcd09-e958-4cbb-9881-c6e6488cb39d"). InnerVolumeSpecName "kube-api-access-kc6w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:09.446654 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.446491 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-tls-certs" pod "8b0fcd09-e958-4cbb-9881-c6e6488cb39d" (UID: "8b0fcd09-e958-4cbb-9881-c6e6488cb39d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:09.545518 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.545474 5084 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-tls-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.545518 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.545519 5084 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-dshm\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.545804 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.545536 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc6w6\" (UniqueName: \"kubernetes.io/projected/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-kube-api-access-kc6w6\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:09.545804 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:09.545550 5084 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0fcd09-e958-4cbb-9881-c6e6488cb39d-home\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:10.246150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:10.246120 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" Nov 03 22:52:10.246637 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:10.246119 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7" event={"ID":"8b0fcd09-e958-4cbb-9881-c6e6488cb39d","Type":"ContainerDied","Data":"9f779940a871687b82dc5d2e7fc5b7038213f48dfceaa967f33463ab3a0f1c59"} Nov 03 22:52:10.246637 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:10.246273 5084 scope.go:122] "RemoveContainer" containerID="29e4a30937d211502610c2ac9b6d52353c9c9fdd318c5c9ab672944c09770478" Nov 03 22:52:10.413879 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:10.413820 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7"] Nov 03 22:52:10.466473 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:10.466430 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-41bb2b/distinct-2-w3-41bb2b-kserve-846694fc58-2smc7"] Nov 03 22:52:10.629543 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:10.629506 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" path="/var/lib/kubelet/pods/8b0fcd09-e958-4cbb-9881-c6e6488cb39d/volumes" Nov 03 22:52:11.069056 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:11.069012 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl"] Nov 03 22:52:11.069341 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:11.069309 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" podUID="fc3ad7a3-068f-4b7b-a58a-082d9d96c7db" containerName="maas-api" containerID="cri-o://4882d2c6a3b55ea588e8a391a3ae005ff63c9caaac270a638c5324dc8a9d0a14" gracePeriod=30 Nov 03 22:52:11.272275 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:11.272234 5084 generic.go:356] "Generic (PLEG): container finished" podID="fc3ad7a3-068f-4b7b-a58a-082d9d96c7db" containerID="4882d2c6a3b55ea588e8a391a3ae005ff63c9caaac270a638c5324dc8a9d0a14" exitCode=0 Nov 03 22:52:11.272778 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:11.272303 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" event={"ID":"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db","Type":"ContainerDied","Data":"4882d2c6a3b55ea588e8a391a3ae005ff63c9caaac270a638c5324dc8a9d0a14"} Nov 03 22:52:11.371482 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:11.371449 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:52:11.471379 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:11.471337 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-metrics-certs\") pod \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\" (UID: \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\") " Nov 03 22:52:11.471593 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:11.471465 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-maas-api-tls\") pod \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\" (UID: \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\") " Nov 03 22:52:11.471593 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:11.471554 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-openshift-service-ca\") pod \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\" (UID: \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\") " Nov 03 22:52:11.471712 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:11.471618 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-kube-api-access-86dr4\" (UniqueName: \"kubernetes.io/projected/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-kube-api-access-86dr4\") pod \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\" (UID: \"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db\") " Nov 03 22:52:11.471987 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:11.471946 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-openshift-service-ca" pod "fc3ad7a3-068f-4b7b-a58a-082d9d96c7db" (UID: "fc3ad7a3-068f-4b7b-a58a-082d9d96c7db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:11.473784 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:11.473735 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-maas-api-tls" pod "fc3ad7a3-068f-4b7b-a58a-082d9d96c7db" (UID: "fc3ad7a3-068f-4b7b-a58a-082d9d96c7db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:11.473917 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:11.473818 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-metrics-certs" pod "fc3ad7a3-068f-4b7b-a58a-082d9d96c7db" (UID: "fc3ad7a3-068f-4b7b-a58a-082d9d96c7db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:11.474165 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:11.474148 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-kube-api-access-86dr4" pod "fc3ad7a3-068f-4b7b-a58a-082d9d96c7db" (UID: "fc3ad7a3-068f-4b7b-a58a-082d9d96c7db"). InnerVolumeSpecName "kube-api-access-86dr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:11.572758 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:11.572716 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:11.572758 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:11.572750 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:11.572758 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:11.572760 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:11.572758 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:11.572773 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86dr4\" (UniqueName: \"kubernetes.io/projected/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db-kube-api-access-86dr4\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.032352 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:52:12.032317 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:52:12.033079 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.033045 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:52:12.201244 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.201198 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa2863b4-3169-423e-a0ee-65fd49117624-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2863b4-3169-423e-a0ee-65fd49117624-istiod-ca-cert\") pod \"aa2863b4-3169-423e-a0ee-65fd49117624\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " Nov 03 22:52:12.201458 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.201280 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-istio-envoy\") pod \"aa2863b4-3169-423e-a0ee-65fd49117624\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " Nov 03 22:52:12.201458 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.201363 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2863b4-3169-423e-a0ee-65fd49117624-istio-token\" (UniqueName: \"kubernetes.io/projected/aa2863b4-3169-423e-a0ee-65fd49117624-istio-token\") pod \"aa2863b4-3169-423e-a0ee-65fd49117624\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " Nov 03 22:52:12.201458 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.201404 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-workload-socket\") pod \"aa2863b4-3169-423e-a0ee-65fd49117624\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " Nov 03 22:52:12.201458 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.201444 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa2863b4-3169-423e-a0ee-65fd49117624-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2863b4-3169-423e-a0ee-65fd49117624-istio-podinfo\") pod \"aa2863b4-3169-423e-a0ee-65fd49117624\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " Nov 03 22:52:12.201719 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.201482 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2863b4-3169-423e-a0ee-65fd49117624-kube-api-access-zs28v\" (UniqueName: \"kubernetes.io/projected/aa2863b4-3169-423e-a0ee-65fd49117624-kube-api-access-zs28v\") pod \"aa2863b4-3169-423e-a0ee-65fd49117624\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " Nov 03 22:52:12.201719 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.201549 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-workload-certs\") pod \"aa2863b4-3169-423e-a0ee-65fd49117624\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " Nov 03 22:52:12.201719 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.201609 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-istio-data\") pod \"aa2863b4-3169-423e-a0ee-65fd49117624\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " Nov 03 22:52:12.201719 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.201606 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2863b4-3169-423e-a0ee-65fd49117624-istiod-ca-cert" pod "aa2863b4-3169-423e-a0ee-65fd49117624" (UID: "aa2863b4-3169-423e-a0ee-65fd49117624"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:12.201719 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.201661 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-credential-socket\") pod \"aa2863b4-3169-423e-a0ee-65fd49117624\" (UID: \"aa2863b4-3169-423e-a0ee-65fd49117624\") " Nov 03 22:52:12.202026 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.201922 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-workload-socket" pod "aa2863b4-3169-423e-a0ee-65fd49117624" (UID: "aa2863b4-3169-423e-a0ee-65fd49117624"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:12.202202 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.202179 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.202298 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.202208 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2863b4-3169-423e-a0ee-65fd49117624-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.203507 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.203473 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-credential-socket" pod "aa2863b4-3169-423e-a0ee-65fd49117624" (UID: "aa2863b4-3169-423e-a0ee-65fd49117624"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:12.203645 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.203552 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-workload-certs" pod "aa2863b4-3169-423e-a0ee-65fd49117624" (UID: "aa2863b4-3169-423e-a0ee-65fd49117624"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:12.203645 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.203562 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-istio-data" pod "aa2863b4-3169-423e-a0ee-65fd49117624" (UID: "aa2863b4-3169-423e-a0ee-65fd49117624"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:12.204703 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.204673 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-istio-envoy" pod "aa2863b4-3169-423e-a0ee-65fd49117624" (UID: "aa2863b4-3169-423e-a0ee-65fd49117624"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:12.206245 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.206211 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa2863b4-3169-423e-a0ee-65fd49117624-istio-podinfo" pod "aa2863b4-3169-423e-a0ee-65fd49117624" (UID: "aa2863b4-3169-423e-a0ee-65fd49117624"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:52:12.206351 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.206214 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2863b4-3169-423e-a0ee-65fd49117624-istio-token" pod "aa2863b4-3169-423e-a0ee-65fd49117624" (UID: "aa2863b4-3169-423e-a0ee-65fd49117624"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:12.206644 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.206577 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2863b4-3169-423e-a0ee-65fd49117624-kube-api-access-zs28v" pod "aa2863b4-3169-423e-a0ee-65fd49117624" (UID: "aa2863b4-3169-423e-a0ee-65fd49117624"). InnerVolumeSpecName "kube-api-access-zs28v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:12.281473 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.281425 5084 generic.go:356] "Generic (PLEG): container finished" podID="aa2863b4-3169-423e-a0ee-65fd49117624" containerID="60f124f6b353f0ae2aa2400e506f6441d968f2dfea4df98c3f0e425bae44889b" exitCode=0 Nov 03 22:52:12.281951 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.281467 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" event={"ID":"aa2863b4-3169-423e-a0ee-65fd49117624","Type":"ContainerDied","Data":"60f124f6b353f0ae2aa2400e506f6441d968f2dfea4df98c3f0e425bae44889b"} Nov 03 22:52:12.281951 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.281534 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" event={"ID":"aa2863b4-3169-423e-a0ee-65fd49117624","Type":"ContainerDied","Data":"cd59f94c7ccbf2e638520d36c620c8e63be9e892cecf2918cab40584d17fe33d"} Nov 03 22:52:12.281951 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.281558 5084 scope.go:122] "RemoveContainer" containerID="60f124f6b353f0ae2aa2400e506f6441d968f2dfea4df98c3f0e425bae44889b" Nov 03 22:52:12.281951 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.281482 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh" Nov 03 22:52:12.285779 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.285749 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" Nov 03 22:52:12.285779 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.285749 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl" event={"ID":"fc3ad7a3-068f-4b7b-a58a-082d9d96c7db","Type":"ContainerDied","Data":"f654694685de7e85b98ebab9d511e3f9c6eb197540eb84b4625e93b47d73e578"} Nov 03 22:52:12.295253 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.295228 5084 scope.go:122] "RemoveContainer" containerID="0523d38e5590c7a26a1500ce2c49cd56aabead8e7c904c4a21a66f807c60e87e" Nov 03 22:52:12.302833 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.302794 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.302833 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.302825 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2863b4-3169-423e-a0ee-65fd49117624-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.302833 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.302837 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2863b4-3169-423e-a0ee-65fd49117624-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.303180 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.302846 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs28v\" (UniqueName: \"kubernetes.io/projected/aa2863b4-3169-423e-a0ee-65fd49117624-kube-api-access-zs28v\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.303180 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.302856 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.303180 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.302865 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.303180 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.302874 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2863b4-3169-423e-a0ee-65fd49117624-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:12.306194 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.306172 5084 scope.go:122] "RemoveContainer" containerID="60f124f6b353f0ae2aa2400e506f6441d968f2dfea4df98c3f0e425bae44889b" Nov 03 22:52:12.306532 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:52:12.306511 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f124f6b353f0ae2aa2400e506f6441d968f2dfea4df98c3f0e425bae44889b\": container with ID starting with 60f124f6b353f0ae2aa2400e506f6441d968f2dfea4df98c3f0e425bae44889b not found: ID does not exist" containerID="60f124f6b353f0ae2aa2400e506f6441d968f2dfea4df98c3f0e425bae44889b" Nov 03 22:52:12.306620 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.306545 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f124f6b353f0ae2aa2400e506f6441d968f2dfea4df98c3f0e425bae44889b"} err="failed to get container status \"60f124f6b353f0ae2aa2400e506f6441d968f2dfea4df98c3f0e425bae44889b\": rpc error: code = NotFound desc = could not find container \"60f124f6b353f0ae2aa2400e506f6441d968f2dfea4df98c3f0e425bae44889b\": container with ID starting with 60f124f6b353f0ae2aa2400e506f6441d968f2dfea4df98c3f0e425bae44889b not found: ID does not exist" Nov 03 22:52:12.306620 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.306564 5084 scope.go:122] "RemoveContainer" containerID="0523d38e5590c7a26a1500ce2c49cd56aabead8e7c904c4a21a66f807c60e87e" Nov 03 22:52:12.306840 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:52:12.306823 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0523d38e5590c7a26a1500ce2c49cd56aabead8e7c904c4a21a66f807c60e87e\": container with ID starting with 0523d38e5590c7a26a1500ce2c49cd56aabead8e7c904c4a21a66f807c60e87e not found: ID does not exist" containerID="0523d38e5590c7a26a1500ce2c49cd56aabead8e7c904c4a21a66f807c60e87e" Nov 03 22:52:12.306882 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.306847 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0523d38e5590c7a26a1500ce2c49cd56aabead8e7c904c4a21a66f807c60e87e"} err="failed to get container status \"0523d38e5590c7a26a1500ce2c49cd56aabead8e7c904c4a21a66f807c60e87e\": rpc error: code = NotFound desc = could not find container \"0523d38e5590c7a26a1500ce2c49cd56aabead8e7c904c4a21a66f807c60e87e\": container with ID starting with 0523d38e5590c7a26a1500ce2c49cd56aabead8e7c904c4a21a66f807c60e87e not found: ID does not exist" Nov 03 22:52:12.306882 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.306862 5084 scope.go:122] "RemoveContainer" containerID="4882d2c6a3b55ea588e8a391a3ae005ff63c9caaac270a638c5324dc8a9d0a14" Nov 03 22:52:12.449484 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.449398 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl"] Nov 03 22:52:12.454570 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.454536 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b30a14-96959c44b-7vcpl"] Nov 03 22:52:12.600246 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.600210 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh"] Nov 03 22:52:12.604081 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.604030 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f61714-openshift-default-d8dbd5bf5-9fmvh"] Nov 03 22:52:12.630568 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.630536 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa2863b4-3169-423e-a0ee-65fd49117624" path="/var/lib/kubelet/pods/aa2863b4-3169-423e-a0ee-65fd49117624/volumes" Nov 03 22:52:12.631045 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:12.631021 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc3ad7a3-068f-4b7b-a58a-082d9d96c7db" path="/var/lib/kubelet/pods/fc3ad7a3-068f-4b7b-a58a-082d9d96c7db/volumes" Nov 03 22:52:13.151357 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:13.151326 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:52:13.152584 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:13.152556 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:52:13.585626 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:13.585565 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj"] Nov 03 22:52:13.636503 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:13.636464 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq"] Nov 03 22:52:20.761578 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:20.761533 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx"] Nov 03 22:52:25.763431 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:25.763370 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" podUID="1a5c9a91-01a9-40e6-80f0-daa26a323c21" containerName="payload-processing" containerID="cri-o://a8c83a5d73d3d730c6c6fdb86e513b1a1a42c3b2061d5000817cfab2e681dccd" gracePeriod=25 Nov 03 22:52:25.784477 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:52:25.784447 5084 logging.go:55] [core] [Channel #4050 SubChannel #4051]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.60:50052", ServerName: "10.132.0.60:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.60:50052: connect: connection refused" Nov 03 22:52:26.037789 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:26.037745 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" Nov 03 22:52:26.105309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:26.105269 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a5c9a91-01a9-40e6-80f0-daa26a323c21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a5c9a91-01a9-40e6-80f0-daa26a323c21-plugins-config-volume\") pod \"1a5c9a91-01a9-40e6-80f0-daa26a323c21\" (UID: \"1a5c9a91-01a9-40e6-80f0-daa26a323c21\") " Nov 03 22:52:26.105503 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:26.105365 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a5c9a91-01a9-40e6-80f0-daa26a323c21-kube-api-access-kclt9\" (UniqueName: \"kubernetes.io/projected/1a5c9a91-01a9-40e6-80f0-daa26a323c21-kube-api-access-kclt9\") pod \"1a5c9a91-01a9-40e6-80f0-daa26a323c21\" (UID: \"1a5c9a91-01a9-40e6-80f0-daa26a323c21\") " Nov 03 22:52:26.105679 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:26.105655 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a5c9a91-01a9-40e6-80f0-daa26a323c21-plugins-config-volume" pod "1a5c9a91-01a9-40e6-80f0-daa26a323c21" (UID: "1a5c9a91-01a9-40e6-80f0-daa26a323c21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:26.105788 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:26.105764 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a5c9a91-01a9-40e6-80f0-daa26a323c21-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:26.107682 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:26.107654 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5c9a91-01a9-40e6-80f0-daa26a323c21-kube-api-access-kclt9" pod "1a5c9a91-01a9-40e6-80f0-daa26a323c21" (UID: "1a5c9a91-01a9-40e6-80f0-daa26a323c21"). InnerVolumeSpecName "kube-api-access-kclt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:26.206704 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:26.206656 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kclt9\" (UniqueName: \"kubernetes.io/projected/1a5c9a91-01a9-40e6-80f0-daa26a323c21-kube-api-access-kclt9\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:26.400182 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:26.400038 5084 generic.go:356] "Generic (PLEG): container finished" podID="1a5c9a91-01a9-40e6-80f0-daa26a323c21" containerID="a8c83a5d73d3d730c6c6fdb86e513b1a1a42c3b2061d5000817cfab2e681dccd" exitCode=0 Nov 03 22:52:26.400182 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:26.400098 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" event={"ID":"1a5c9a91-01a9-40e6-80f0-daa26a323c21","Type":"ContainerDied","Data":"a8c83a5d73d3d730c6c6fdb86e513b1a1a42c3b2061d5000817cfab2e681dccd"} Nov 03 22:52:26.400182 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:26.400136 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" Nov 03 22:52:26.400182 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:26.400173 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" event={"ID":"1a5c9a91-01a9-40e6-80f0-daa26a323c21","Type":"ContainerDied","Data":"9e7ff5fff18f8ac631bd1d03c454336f0755f752fb985c34756cc7dd01affebc"} Nov 03 22:52:26.400553 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:26.400204 5084 scope.go:122] "RemoveContainer" containerID="a8c83a5d73d3d730c6c6fdb86e513b1a1a42c3b2061d5000817cfab2e681dccd" Nov 03 22:52:26.412499 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:26.412475 5084 scope.go:122] "RemoveContainer" containerID="a8c83a5d73d3d730c6c6fdb86e513b1a1a42c3b2061d5000817cfab2e681dccd" Nov 03 22:52:26.412759 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:52:26.412737 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c83a5d73d3d730c6c6fdb86e513b1a1a42c3b2061d5000817cfab2e681dccd\": container with ID starting with a8c83a5d73d3d730c6c6fdb86e513b1a1a42c3b2061d5000817cfab2e681dccd not found: ID does not exist" containerID="a8c83a5d73d3d730c6c6fdb86e513b1a1a42c3b2061d5000817cfab2e681dccd" Nov 03 22:52:26.412833 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:26.412768 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c83a5d73d3d730c6c6fdb86e513b1a1a42c3b2061d5000817cfab2e681dccd"} err="failed to get container status \"a8c83a5d73d3d730c6c6fdb86e513b1a1a42c3b2061d5000817cfab2e681dccd\": rpc error: code = NotFound desc = could not find container \"a8c83a5d73d3d730c6c6fdb86e513b1a1a42c3b2061d5000817cfab2e681dccd\": container with ID starting with a8c83a5d73d3d730c6c6fdb86e513b1a1a42c3b2061d5000817cfab2e681dccd not found: ID does not exist" Nov 03 22:52:26.568453 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:26.568411 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx"] Nov 03 22:52:26.619294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:26.619254 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx"] Nov 03 22:52:26.630382 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:26.630345 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a5c9a91-01a9-40e6-80f0-daa26a323c21" path="/var/lib/kubelet/pods/1a5c9a91-01a9-40e6-80f0-daa26a323c21/volumes" Nov 03 22:52:26.784594 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:26.784547 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-f7e6db-54898964d6-9vdsx" podUID="1a5c9a91-01a9-40e6-80f0-daa26a323c21" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.60:50052\" within 1s: context deadline exceeded" Nov 03 22:52:28.607681 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:28.607631 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" containerID="cri-o://058da7f6b7bb7eb57aee663e4737ef42f0190373218377565b86077d2f779d09" gracePeriod=45 Nov 03 22:52:28.657758 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:28.657712 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" containerID="cri-o://0aca803caaefa0066cc726e6bce7b3a460febd46ef87037b3f01ea39d297c1ed" gracePeriod=45 Nov 03 22:52:28.723520 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:28.723478 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": dial tcp 10.132.0.71:8000: connect: connection refused" Nov 03 22:52:28.752416 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:28.752368 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": dial tcp 10.132.0.78:8000: connect: connection refused" Nov 03 22:52:28.939152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:28.939123 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:52:28.974548 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:28.974510 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:52:29.040955 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.040919 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-model-cache\") pod \"8de619f3-ea48-44dc-b8a6-d70502f99760\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " Nov 03 22:52:29.041145 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.040970 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-tmp-dir\") pod \"8de619f3-ea48-44dc-b8a6-d70502f99760\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " Nov 03 22:52:29.041145 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.041002 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-dshm\") pod \"8de619f3-ea48-44dc-b8a6-d70502f99760\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " Nov 03 22:52:29.041145 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.041124 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-home\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-home\") pod \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " Nov 03 22:52:29.041358 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.041299 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-model-cache" pod "8de619f3-ea48-44dc-b8a6-d70502f99760" (UID: "8de619f3-ea48-44dc-b8a6-d70502f99760"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:29.041429 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.041379 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-tmp-dir" pod "8de619f3-ea48-44dc-b8a6-d70502f99760" (UID: "8de619f3-ea48-44dc-b8a6-d70502f99760"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:29.041491 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.041444 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-kube-api-access-vvxgq\" (UniqueName: \"kubernetes.io/projected/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-kube-api-access-vvxgq\") pod \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " Nov 03 22:52:29.041609 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.041573 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-home\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-home\") pod \"8de619f3-ea48-44dc-b8a6-d70502f99760\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " Nov 03 22:52:29.041856 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.041579 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-home" pod "ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" (UID: "ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:29.041983 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.041842 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-model-cache\") pod \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " Nov 03 22:52:29.041983 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.041786 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-home" pod "8de619f3-ea48-44dc-b8a6-d70502f99760" (UID: "8de619f3-ea48-44dc-b8a6-d70502f99760"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:29.041983 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.041897 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-dshm\") pod \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " Nov 03 22:52:29.041983 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.041941 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de619f3-ea48-44dc-b8a6-d70502f99760-kube-api-access-fl8nl\" (UniqueName: \"kubernetes.io/projected/8de619f3-ea48-44dc-b8a6-d70502f99760-kube-api-access-fl8nl\") pod \"8de619f3-ea48-44dc-b8a6-d70502f99760\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " Nov 03 22:52:29.042264 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.042005 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8de619f3-ea48-44dc-b8a6-d70502f99760-tls-certs\" (UniqueName: \"kubernetes.io/secret/8de619f3-ea48-44dc-b8a6-d70502f99760-tls-certs\") pod \"8de619f3-ea48-44dc-b8a6-d70502f99760\" (UID: \"8de619f3-ea48-44dc-b8a6-d70502f99760\") " Nov 03 22:52:29.042264 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.042033 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-tmp-dir\") pod \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " Nov 03 22:52:29.042264 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.042043 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-model-cache" pod "ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" (UID: "ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:29.042264 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.042169 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-tls-certs\") pod \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\" (UID: \"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49\") " Nov 03 22:52:29.042647 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.042605 5084 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-home\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.042647 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.042633 5084 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-model-cache\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.042796 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.042648 5084 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-model-cache\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.042796 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.042663 5084 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-tmp-dir\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.042796 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.042677 5084 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-home\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.042998 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.042977 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-tmp-dir" pod "ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" (UID: "ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:29.044265 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.044238 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-dshm" pod "8de619f3-ea48-44dc-b8a6-d70502f99760" (UID: "8de619f3-ea48-44dc-b8a6-d70502f99760"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:29.044265 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.044249 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-kube-api-access-vvxgq" pod "ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" (UID: "ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49"). InnerVolumeSpecName "kube-api-access-vvxgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:29.044470 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.044320 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de619f3-ea48-44dc-b8a6-d70502f99760-tls-certs" pod "8de619f3-ea48-44dc-b8a6-d70502f99760" (UID: "8de619f3-ea48-44dc-b8a6-d70502f99760"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:29.044563 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.044543 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-dshm" pod "ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" (UID: "ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:29.044735 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.044714 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-tls-certs" pod "ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" (UID: "ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:29.045790 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.045758 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de619f3-ea48-44dc-b8a6-d70502f99760-kube-api-access-fl8nl" pod "8de619f3-ea48-44dc-b8a6-d70502f99760" (UID: "8de619f3-ea48-44dc-b8a6-d70502f99760"). InnerVolumeSpecName "kube-api-access-fl8nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:29.143538 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.143491 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvxgq\" (UniqueName: \"kubernetes.io/projected/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-kube-api-access-vvxgq\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.143538 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.143533 5084 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-dshm\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.143538 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.143548 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl8nl\" (UniqueName: \"kubernetes.io/projected/8de619f3-ea48-44dc-b8a6-d70502f99760-kube-api-access-fl8nl\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.143797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.143563 5084 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de619f3-ea48-44dc-b8a6-d70502f99760-tls-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.143797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.143575 5084 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-tmp-dir\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.143797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.143586 5084 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49-tls-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.143797 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.143597 5084 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de619f3-ea48-44dc-b8a6-d70502f99760-dshm\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:29.422638 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.422592 5084 generic.go:356] "Generic (PLEG): container finished" podID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerID="0aca803caaefa0066cc726e6bce7b3a460febd46ef87037b3f01ea39d297c1ed" exitCode=0 Nov 03 22:52:29.422862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.422674 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" Nov 03 22:52:29.422862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.422677 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" event={"ID":"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49","Type":"ContainerDied","Data":"0aca803caaefa0066cc726e6bce7b3a460febd46ef87037b3f01ea39d297c1ed"} Nov 03 22:52:29.422862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.422765 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq" event={"ID":"ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49","Type":"ContainerDied","Data":"6afbfcd33dbd98326a1d5fea5b959548ddbf4bc1bb326889ff70237e176d2ad3"} Nov 03 22:52:29.422862 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.422818 5084 scope.go:122] "RemoveContainer" containerID="0aca803caaefa0066cc726e6bce7b3a460febd46ef87037b3f01ea39d297c1ed" Nov 03 22:52:29.424767 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.424740 5084 generic.go:356] "Generic (PLEG): container finished" podID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerID="058da7f6b7bb7eb57aee663e4737ef42f0190373218377565b86077d2f779d09" exitCode=0 Nov 03 22:52:29.424894 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.424781 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" event={"ID":"8de619f3-ea48-44dc-b8a6-d70502f99760","Type":"ContainerDied","Data":"058da7f6b7bb7eb57aee663e4737ef42f0190373218377565b86077d2f779d09"} Nov 03 22:52:29.424894 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.424808 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" event={"ID":"8de619f3-ea48-44dc-b8a6-d70502f99760","Type":"ContainerDied","Data":"a7fa086bb7f2b8a9a02ce1f31d98cf069c3fb82c65a2d1f67ce823e1686a4b00"} Nov 03 22:52:29.424894 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.424875 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj" Nov 03 22:52:29.433529 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.433496 5084 scope.go:122] "RemoveContainer" containerID="0aca803caaefa0066cc726e6bce7b3a460febd46ef87037b3f01ea39d297c1ed" Nov 03 22:52:29.433906 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:52:29.433877 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aca803caaefa0066cc726e6bce7b3a460febd46ef87037b3f01ea39d297c1ed\": container with ID starting with 0aca803caaefa0066cc726e6bce7b3a460febd46ef87037b3f01ea39d297c1ed not found: ID does not exist" containerID="0aca803caaefa0066cc726e6bce7b3a460febd46ef87037b3f01ea39d297c1ed" Nov 03 22:52:29.433990 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.433917 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aca803caaefa0066cc726e6bce7b3a460febd46ef87037b3f01ea39d297c1ed"} err="failed to get container status \"0aca803caaefa0066cc726e6bce7b3a460febd46ef87037b3f01ea39d297c1ed\": rpc error: code = NotFound desc = could not find container \"0aca803caaefa0066cc726e6bce7b3a460febd46ef87037b3f01ea39d297c1ed\": container with ID starting with 0aca803caaefa0066cc726e6bce7b3a460febd46ef87037b3f01ea39d297c1ed not found: ID does not exist" Nov 03 22:52:29.433990 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.433943 5084 scope.go:122] "RemoveContainer" containerID="058da7f6b7bb7eb57aee663e4737ef42f0190373218377565b86077d2f779d09" Nov 03 22:52:29.443932 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.443907 5084 scope.go:122] "RemoveContainer" containerID="058da7f6b7bb7eb57aee663e4737ef42f0190373218377565b86077d2f779d09" Nov 03 22:52:29.444301 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:52:29.444279 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"058da7f6b7bb7eb57aee663e4737ef42f0190373218377565b86077d2f779d09\": container with ID starting with 058da7f6b7bb7eb57aee663e4737ef42f0190373218377565b86077d2f779d09 not found: ID does not exist" containerID="058da7f6b7bb7eb57aee663e4737ef42f0190373218377565b86077d2f779d09" Nov 03 22:52:29.444379 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.444310 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"058da7f6b7bb7eb57aee663e4737ef42f0190373218377565b86077d2f779d09"} err="failed to get container status \"058da7f6b7bb7eb57aee663e4737ef42f0190373218377565b86077d2f779d09\": rpc error: code = NotFound desc = could not find container \"058da7f6b7bb7eb57aee663e4737ef42f0190373218377565b86077d2f779d09\": container with ID starting with 058da7f6b7bb7eb57aee663e4737ef42f0190373218377565b86077d2f779d09 not found: ID does not exist" Nov 03 22:52:29.590027 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.589989 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj"] Nov 03 22:52:29.593693 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.593660 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/distinct-2-w4-f7e6db-kserve-64db65cf84-tchdj"] Nov 03 22:52:29.704738 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.704653 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq"] Nov 03 22:52:29.711150 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:29.711115 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f7e6db/distinct-w4-f7e6db-kserve-74db565c5-8hnwq"] Nov 03 22:52:30.629551 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:30.629514 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" path="/var/lib/kubelet/pods/8de619f3-ea48-44dc-b8a6-d70502f99760/volumes" Nov 03 22:52:30.630212 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:30.630189 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" path="/var/lib/kubelet/pods/ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49/volumes" Nov 03 22:52:33.395093 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:33.395043 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t"] Nov 03 22:52:33.395536 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:33.395266 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" podUID="1ef54935-c879-44c3-b864-fff0bbfbcfd9" containerName="maas-api" containerID="cri-o://b8b308609580eefd97e5529b2033b7c55d29f183b1db780ae7ddd362f0f29158" gracePeriod=30 Nov 03 22:52:33.690075 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:33.690024 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:52:33.830076 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:33.830023 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ef54935-c879-44c3-b864-fff0bbfbcfd9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ef54935-c879-44c3-b864-fff0bbfbcfd9-maas-api-tls\") pod \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\" (UID: \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\") " Nov 03 22:52:33.830260 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:33.830166 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef54935-c879-44c3-b864-fff0bbfbcfd9-kube-api-access-54rcc\" (UniqueName: \"kubernetes.io/projected/1ef54935-c879-44c3-b864-fff0bbfbcfd9-kube-api-access-54rcc\") pod \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\" (UID: \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\") " Nov 03 22:52:33.830260 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:33.830200 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ef54935-c879-44c3-b864-fff0bbfbcfd9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ef54935-c879-44c3-b864-fff0bbfbcfd9-metrics-certs\") pod \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\" (UID: \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\") " Nov 03 22:52:33.830403 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:33.830258 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef54935-c879-44c3-b864-fff0bbfbcfd9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef54935-c879-44c3-b864-fff0bbfbcfd9-openshift-service-ca\") pod \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\" (UID: \"1ef54935-c879-44c3-b864-fff0bbfbcfd9\") " Nov 03 22:52:33.830889 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:33.830831 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef54935-c879-44c3-b864-fff0bbfbcfd9-openshift-service-ca" pod "1ef54935-c879-44c3-b864-fff0bbfbcfd9" (UID: "1ef54935-c879-44c3-b864-fff0bbfbcfd9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:33.833255 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:33.833210 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ef54935-c879-44c3-b864-fff0bbfbcfd9-metrics-certs" pod "1ef54935-c879-44c3-b864-fff0bbfbcfd9" (UID: "1ef54935-c879-44c3-b864-fff0bbfbcfd9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:33.835613 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:33.835559 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ef54935-c879-44c3-b864-fff0bbfbcfd9-maas-api-tls" pod "1ef54935-c879-44c3-b864-fff0bbfbcfd9" (UID: "1ef54935-c879-44c3-b864-fff0bbfbcfd9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:33.835879 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:33.835851 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef54935-c879-44c3-b864-fff0bbfbcfd9-kube-api-access-54rcc" pod "1ef54935-c879-44c3-b864-fff0bbfbcfd9" (UID: "1ef54935-c879-44c3-b864-fff0bbfbcfd9"). InnerVolumeSpecName "kube-api-access-54rcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:33.931998 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:33.931906 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef54935-c879-44c3-b864-fff0bbfbcfd9-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:33.931998 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:33.931943 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ef54935-c879-44c3-b864-fff0bbfbcfd9-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:33.931998 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:33.931953 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54rcc\" (UniqueName: \"kubernetes.io/projected/1ef54935-c879-44c3-b864-fff0bbfbcfd9-kube-api-access-54rcc\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:33.931998 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:33.931966 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ef54935-c879-44c3-b864-fff0bbfbcfd9-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:34.470177 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:34.470140 5084 generic.go:356] "Generic (PLEG): container finished" podID="1ef54935-c879-44c3-b864-fff0bbfbcfd9" containerID="b8b308609580eefd97e5529b2033b7c55d29f183b1db780ae7ddd362f0f29158" exitCode=0 Nov 03 22:52:34.470667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:34.470204 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" event={"ID":"1ef54935-c879-44c3-b864-fff0bbfbcfd9","Type":"ContainerDied","Data":"b8b308609580eefd97e5529b2033b7c55d29f183b1db780ae7ddd362f0f29158"} Nov 03 22:52:34.470667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:34.470210 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" Nov 03 22:52:34.470667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:34.470245 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t" event={"ID":"1ef54935-c879-44c3-b864-fff0bbfbcfd9","Type":"ContainerDied","Data":"c73016da52ae42d7f885a22441c6bcd3f931980fab7d5fe8249826eff0ac1c5c"} Nov 03 22:52:34.470667 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:34.470267 5084 scope.go:122] "RemoveContainer" containerID="b8b308609580eefd97e5529b2033b7c55d29f183b1db780ae7ddd362f0f29158" Nov 03 22:52:34.495428 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:34.495399 5084 scope.go:122] "RemoveContainer" containerID="b8b308609580eefd97e5529b2033b7c55d29f183b1db780ae7ddd362f0f29158" Nov 03 22:52:34.495866 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:52:34.495840 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b308609580eefd97e5529b2033b7c55d29f183b1db780ae7ddd362f0f29158\": container with ID starting with b8b308609580eefd97e5529b2033b7c55d29f183b1db780ae7ddd362f0f29158 not found: ID does not exist" containerID="b8b308609580eefd97e5529b2033b7c55d29f183b1db780ae7ddd362f0f29158" Nov 03 22:52:34.495979 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:34.495872 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b308609580eefd97e5529b2033b7c55d29f183b1db780ae7ddd362f0f29158"} err="failed to get container status \"b8b308609580eefd97e5529b2033b7c55d29f183b1db780ae7ddd362f0f29158\": rpc error: code = NotFound desc = could not find container \"b8b308609580eefd97e5529b2033b7c55d29f183b1db780ae7ddd362f0f29158\": container with ID starting with b8b308609580eefd97e5529b2033b7c55d29f183b1db780ae7ddd362f0f29158 not found: ID does not exist" Nov 03 22:52:34.639245 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:34.639205 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t"] Nov 03 22:52:34.690279 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:34.690239 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-fac3b6-d47c58986-b2v2t"] Nov 03 22:52:36.628228 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:36.628194 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef54935-c879-44c3-b864-fff0bbfbcfd9" path="/var/lib/kubelet/pods/1ef54935-c879-44c3-b864-fff0bbfbcfd9/volumes" Nov 03 22:52:42.024223 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:52:42.024175 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:52:46.280406 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.280368 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k"] Nov 03 22:52:46.281116 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281088 5084 cpu_manager.go:414] "containerMap: removing container" podUID="f2ea5507-0676-4e91-a1fd-b13b524cd473" containerName="maas-api" Nov 03 22:52:46.281116 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281116 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ea5507-0676-4e91-a1fd-b13b524cd473" containerName="maas-api" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281138 5084 cpu_manager.go:414] "containerMap: removing container" podUID="1a5c9a91-01a9-40e6-80f0-daa26a323c21" containerName="payload-processing" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281146 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a5c9a91-01a9-40e6-80f0-daa26a323c21" containerName="payload-processing" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281159 5084 cpu_manager.go:414] "containerMap: removing container" podUID="f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" containerName="istio-proxy" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281168 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" containerName="istio-proxy" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281179 5084 cpu_manager.go:414] "containerMap: removing container" podUID="912af5f3-b029-4368-b74b-693937c9431d" containerName="maas-api" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281186 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="912af5f3-b029-4368-b74b-693937c9431d" containerName="maas-api" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281199 5084 cpu_manager.go:414] "containerMap: removing container" podUID="1ef54935-c879-44c3-b864-fff0bbfbcfd9" containerName="maas-api" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281206 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef54935-c879-44c3-b864-fff0bbfbcfd9" containerName="maas-api" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281219 5084 cpu_manager.go:414] "containerMap: removing container" podUID="892a000a-c16a-4904-8d6d-57952348afc9" containerName="istio-proxy" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281226 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892a000a-c16a-4904-8d6d-57952348afc9" containerName="istio-proxy" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281242 5084 cpu_manager.go:414] "containerMap: removing container" podUID="135fb3ee-0298-424f-b188-611bdfbec980" containerName="istio-proxy" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281251 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135fb3ee-0298-424f-b188-611bdfbec980" containerName="istio-proxy" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281261 5084 cpu_manager.go:414] "containerMap: removing container" podUID="49089888-27d7-482d-a915-450990214859" containerName="main" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281268 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49089888-27d7-482d-a915-450990214859" containerName="main" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281282 5084 cpu_manager.go:414] "containerMap: removing container" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281289 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281300 5084 cpu_manager.go:414] "containerMap: removing container" podUID="aa2863b4-3169-423e-a0ee-65fd49117624" containerName="istio-proxy" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281307 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2863b4-3169-423e-a0ee-65fd49117624" containerName="istio-proxy" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281317 5084 cpu_manager.go:414] "containerMap: removing container" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281327 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" Nov 03 22:52:46.281323 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281337 5084 cpu_manager.go:414] "containerMap: removing container" podUID="2da4671a-8a4b-4ce4-9eb7-81183fe89ad6" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281344 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da4671a-8a4b-4ce4-9eb7-81183fe89ad6" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281354 5084 cpu_manager.go:414] "containerMap: removing container" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281361 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281369 5084 cpu_manager.go:414] "containerMap: removing container" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281376 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281389 5084 cpu_manager.go:414] "containerMap: removing container" podUID="3956759d-6df8-4ae5-b9db-80933d555c9c" containerName="payload-pre-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281396 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3956759d-6df8-4ae5-b9db-80933d555c9c" containerName="payload-pre-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281407 5084 cpu_manager.go:414] "containerMap: removing container" podUID="ae836564-7d17-4919-8a65-f655fabe9759" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281413 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae836564-7d17-4919-8a65-f655fabe9759" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281428 5084 cpu_manager.go:414] "containerMap: removing container" podUID="b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281433 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281440 5084 cpu_manager.go:414] "containerMap: removing container" podUID="e533f5a5-fd74-4be2-828b-5efd065720ff" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281444 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e533f5a5-fd74-4be2-828b-5efd065720ff" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281453 5084 cpu_manager.go:414] "containerMap: removing container" podUID="429389e2-f479-4c31-9170-bf11ff231d48" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281461 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="429389e2-f479-4c31-9170-bf11ff231d48" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281471 5084 cpu_manager.go:414] "containerMap: removing container" podUID="9fbbe6f8-7037-43d3-b3e3-0e48b6316632" containerName="payload-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281476 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbbe6f8-7037-43d3-b3e3-0e48b6316632" containerName="payload-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281483 5084 cpu_manager.go:414] "containerMap: removing container" podUID="29c0957d-9d08-4667-b81e-fe1347e4805a" containerName="payload-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281487 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c0957d-9d08-4667-b81e-fe1347e4805a" containerName="payload-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281494 5084 cpu_manager.go:414] "containerMap: removing container" podUID="bb5188a0-6554-4eb7-8567-0613709d03d9" containerName="istio-proxy" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281499 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5188a0-6554-4eb7-8567-0613709d03d9" containerName="istio-proxy" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281506 5084 cpu_manager.go:414] "containerMap: removing container" podUID="e76224de-abc4-45e7-8502-87f35e73f3cb" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281511 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76224de-abc4-45e7-8502-87f35e73f3cb" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281517 5084 cpu_manager.go:414] "containerMap: removing container" podUID="e9dab1e4-6fcb-49a2-b04d-a803633c232d" containerName="payload-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281522 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9dab1e4-6fcb-49a2-b04d-a803633c232d" containerName="payload-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281528 5084 cpu_manager.go:414] "containerMap: removing container" podUID="df48770c-9c54-43d0-87c8-bc7a65bebf2c" containerName="main" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281532 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df48770c-9c54-43d0-87c8-bc7a65bebf2c" containerName="main" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281544 5084 cpu_manager.go:414] "containerMap: removing container" podUID="9a565b6a-0d39-4a6b-bfeb-af467508b071" containerName="payload-pre-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281548 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a565b6a-0d39-4a6b-bfeb-af467508b071" containerName="payload-pre-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281554 5084 cpu_manager.go:414] "containerMap: removing container" podUID="7e55f64c-dc5c-40c8-ae1f-6b864da55c11" containerName="istio-proxy" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281559 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e55f64c-dc5c-40c8-ae1f-6b864da55c11" containerName="istio-proxy" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281567 5084 cpu_manager.go:414] "containerMap: removing container" podUID="be288892-c462-4214-a907-c7a2b7b75c25" containerName="payload-pre-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281574 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be288892-c462-4214-a907-c7a2b7b75c25" containerName="payload-pre-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281591 5084 cpu_manager.go:414] "containerMap: removing container" podUID="4e4d71d8-dfc0-437a-a639-8e763b69f249" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281596 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4d71d8-dfc0-437a-a639-8e763b69f249" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281604 5084 cpu_manager.go:414] "containerMap: removing container" podUID="531b6497-408c-4315-806e-0504eead114a" containerName="istio-proxy" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281608 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="531b6497-408c-4315-806e-0504eead114a" containerName="istio-proxy" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281616 5084 cpu_manager.go:414] "containerMap: removing container" podUID="fc3ad7a3-068f-4b7b-a58a-082d9d96c7db" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281620 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc3ad7a3-068f-4b7b-a58a-082d9d96c7db" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281628 5084 cpu_manager.go:414] "containerMap: removing container" podUID="b94934e9-8f2d-4b49-93f2-a386807c4c79" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281632 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b94934e9-8f2d-4b49-93f2-a386807c4c79" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281745 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4d71d8-dfc0-437a-a639-8e763b69f249" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281761 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="531b6497-408c-4315-806e-0504eead114a" containerName="istio-proxy" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281772 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc3ad7a3-068f-4b7b-a58a-082d9d96c7db" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281788 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="b94934e9-8f2d-4b49-93f2-a386807c4c79" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281797 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ea5507-0676-4e91-a1fd-b13b524cd473" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281804 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a5c9a91-01a9-40e6-80f0-daa26a323c21" containerName="payload-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281813 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" containerName="istio-proxy" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281820 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="912af5f3-b029-4368-b74b-693937c9431d" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281831 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef54935-c879-44c3-b864-fff0bbfbcfd9" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281839 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="135fb3ee-0298-424f-b188-611bdfbec980" containerName="istio-proxy" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281846 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="49089888-27d7-482d-a915-450990214859" containerName="main" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281854 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b0fcd09-e958-4cbb-9881-c6e6488cb39d" containerName="main" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281862 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2863b4-3169-423e-a0ee-65fd49117624" containerName="istio-proxy" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281871 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b28118-7e39-4b66-be5f-f5a61a199f85" containerName="main" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281879 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da4671a-8a4b-4ce4-9eb7-81183fe89ad6" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281889 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca63bdeb-70de-46b0-b8ec-ba3e5a57fa49" containerName="main" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281897 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de619f3-ea48-44dc-b8a6-d70502f99760" containerName="main" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281918 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="3956759d-6df8-4ae5-b9db-80933d555c9c" containerName="payload-pre-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281926 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae836564-7d17-4919-8a65-f655fabe9759" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281936 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1cc5504-a4f0-4f55-91bd-e2bc277fc7a2" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281943 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="e533f5a5-fd74-4be2-828b-5efd065720ff" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281951 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="429389e2-f479-4c31-9170-bf11ff231d48" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281960 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbbe6f8-7037-43d3-b3e3-0e48b6316632" containerName="payload-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281968 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c0957d-9d08-4667-b81e-fe1347e4805a" containerName="payload-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281978 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5188a0-6554-4eb7-8567-0613709d03d9" containerName="istio-proxy" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281986 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2863b4-3169-423e-a0ee-65fd49117624" containerName="istio-proxy" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.281993 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76224de-abc4-45e7-8502-87f35e73f3cb" containerName="maas-api" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.282002 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9dab1e4-6fcb-49a2-b04d-a803633c232d" containerName="payload-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.282011 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="df48770c-9c54-43d0-87c8-bc7a65bebf2c" containerName="main" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.282023 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a565b6a-0d39-4a6b-bfeb-af467508b071" containerName="payload-pre-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.282032 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e55f64c-dc5c-40c8-ae1f-6b864da55c11" containerName="istio-proxy" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.282040 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="be288892-c462-4214-a907-c7a2b7b75c25" containerName="payload-pre-processing" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.282048 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" containerName="istio-proxy" Nov 03 22:52:46.282309 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.282057 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="135fb3ee-0298-424f-b188-611bdfbec980" containerName="istio-proxy" Nov 03 22:52:46.297032 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.293201 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k" Nov 03 22:52:46.336530 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.336490 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k"] Nov 03 22:52:46.366882 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.366835 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4236a363-fkm6k\" (UID: \"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k" Nov 03 22:52:46.366882 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.366883 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4236a363-fkm6k\" (UID: \"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k" Nov 03 22:52:46.367184 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.367052 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ls9k\" (UniqueName: \"kubernetes.io/projected/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-kube-api-access-6ls9k\") pod \"maas-api-revoke-sub-4236a363-fkm6k\" (UID: \"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k" Nov 03 22:52:46.384727 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.384692 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:52:46.384906 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.384730 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:52:46.467734 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.467688 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ls9k\" (UniqueName: \"kubernetes.io/projected/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-kube-api-access-6ls9k\") pod \"maas-api-revoke-sub-4236a363-fkm6k\" (UID: \"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k" Nov 03 22:52:46.467938 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.467774 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4236a363-fkm6k\" (UID: \"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k" Nov 03 22:52:46.467938 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.467793 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4236a363-fkm6k\" (UID: \"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k" Nov 03 22:52:46.468608 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.468576 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4236a363-fkm6k\" (UID: \"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k" Nov 03 22:52:46.470452 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.470424 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4236a363-fkm6k\" (UID: \"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k" Nov 03 22:52:46.525872 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.525835 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ls9k\" (UniqueName: \"kubernetes.io/projected/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-kube-api-access-6ls9k\") pod \"maas-api-revoke-sub-4236a363-fkm6k\" (UID: \"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k" Nov 03 22:52:46.555942 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.555850 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx"] Nov 03 22:52:46.556682 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.556653 5084 cpu_manager.go:414] "containerMap: removing container" podUID="f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" containerName="istio-proxy" Nov 03 22:52:46.556682 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.556681 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2862fdc-74c6-4f14-9a9b-ced8a3d538d5" containerName="istio-proxy" Nov 03 22:52:46.556912 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.556706 5084 cpu_manager.go:414] "containerMap: removing container" podUID="135fb3ee-0298-424f-b188-611bdfbec980" containerName="istio-proxy" Nov 03 22:52:46.556912 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.556715 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135fb3ee-0298-424f-b188-611bdfbec980" containerName="istio-proxy" Nov 03 22:52:46.556912 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.556732 5084 cpu_manager.go:414] "containerMap: removing container" podUID="aa2863b4-3169-423e-a0ee-65fd49117624" containerName="istio-proxy" Nov 03 22:52:46.556912 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.556740 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2863b4-3169-423e-a0ee-65fd49117624" containerName="istio-proxy" Nov 03 22:52:46.561188 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.561154 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx" Nov 03 22:52:46.611279 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.611236 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k" Nov 03 22:52:46.612750 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.612585 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx"] Nov 03 22:52:46.670917 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.670159 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zmx5\" (UniqueName: \"kubernetes.io/projected/fef5ac70-c9c9-46ad-aadb-7f5a54264240-kube-api-access-2zmx5\") pod \"maas-api-revoke-sub-a8d76e57-g22tx\" (UID: \"fef5ac70-c9c9-46ad-aadb-7f5a54264240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx" Nov 03 22:52:46.670917 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.670334 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef5ac70-c9c9-46ad-aadb-7f5a54264240-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8d76e57-g22tx\" (UID: \"fef5ac70-c9c9-46ad-aadb-7f5a54264240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx" Nov 03 22:52:46.670917 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.670425 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef5ac70-c9c9-46ad-aadb-7f5a54264240-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8d76e57-g22tx\" (UID: \"fef5ac70-c9c9-46ad-aadb-7f5a54264240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx" Nov 03 22:52:46.772075 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.772017 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef5ac70-c9c9-46ad-aadb-7f5a54264240-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8d76e57-g22tx\" (UID: \"fef5ac70-c9c9-46ad-aadb-7f5a54264240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx" Nov 03 22:52:46.772301 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.772259 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef5ac70-c9c9-46ad-aadb-7f5a54264240-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8d76e57-g22tx\" (UID: \"fef5ac70-c9c9-46ad-aadb-7f5a54264240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx" Nov 03 22:52:46.772398 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.772376 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zmx5\" (UniqueName: \"kubernetes.io/projected/fef5ac70-c9c9-46ad-aadb-7f5a54264240-kube-api-access-2zmx5\") pod \"maas-api-revoke-sub-a8d76e57-g22tx\" (UID: \"fef5ac70-c9c9-46ad-aadb-7f5a54264240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx" Nov 03 22:52:46.773009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.772985 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef5ac70-c9c9-46ad-aadb-7f5a54264240-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8d76e57-g22tx\" (UID: \"fef5ac70-c9c9-46ad-aadb-7f5a54264240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx" Nov 03 22:52:46.774929 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.774906 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef5ac70-c9c9-46ad-aadb-7f5a54264240-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8d76e57-g22tx\" (UID: \"fef5ac70-c9c9-46ad-aadb-7f5a54264240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx" Nov 03 22:52:46.830545 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.830472 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zmx5\" (UniqueName: \"kubernetes.io/projected/fef5ac70-c9c9-46ad-aadb-7f5a54264240-kube-api-access-2zmx5\") pod \"maas-api-revoke-sub-a8d76e57-g22tx\" (UID: \"fef5ac70-c9c9-46ad-aadb-7f5a54264240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx" Nov 03 22:52:46.881157 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:46.881116 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx" Nov 03 22:52:47.024647 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:47.024613 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k"] Nov 03 22:52:47.026097 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:52:47.026050 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3f9cad_3ed7_440f_b9e3_b3371d649bb8.slice/crio-e1ee7575c3ef1d689ff043a0065e3292da81d0c7d309b084fb1ff5d87cbb1f5e WatchSource:0}: Error finding container e1ee7575c3ef1d689ff043a0065e3292da81d0c7d309b084fb1ff5d87cbb1f5e: Status 404 returned error can't find the container with id e1ee7575c3ef1d689ff043a0065e3292da81d0c7d309b084fb1ff5d87cbb1f5e Nov 03 22:52:47.263546 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:47.263513 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx"] Nov 03 22:52:47.271483 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:52:47.271452 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef5ac70_c9c9_46ad_aadb_7f5a54264240.slice/crio-238db2634caf753da168a9283d331e60b4c11d01f327c2be5639c42b9b28ae21 WatchSource:0}: Error finding container 238db2634caf753da168a9283d331e60b4c11d01f327c2be5639c42b9b28ae21: Status 404 returned error can't find the container with id 238db2634caf753da168a9283d331e60b4c11d01f327c2be5639c42b9b28ae21 Nov 03 22:52:47.554739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:47.554695 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx" event={"ID":"fef5ac70-c9c9-46ad-aadb-7f5a54264240","Type":"ContainerStarted","Data":"238db2634caf753da168a9283d331e60b4c11d01f327c2be5639c42b9b28ae21"} Nov 03 22:52:47.556746 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:47.556715 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k" event={"ID":"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8","Type":"ContainerStarted","Data":"5d86de75a304eb49af29e388a84aa327bc42627394297f07aedcdbd5ddb67533"} Nov 03 22:52:47.557032 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:47.556777 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k" event={"ID":"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8","Type":"ContainerStarted","Data":"e1ee7575c3ef1d689ff043a0065e3292da81d0c7d309b084fb1ff5d87cbb1f5e"} Nov 03 22:52:47.673197 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:47.673131 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k" podStartSLOduration=1.673107838 podStartE2EDuration="1.673107838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:47.671520947 +0000 UTC m=+2735.265552922" watchObservedRunningTime="2026-11-03 22:52:47.673107838 +0000 UTC m=+2735.267139811" Nov 03 22:52:48.564474 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:48.564441 5084 generic.go:356] "Generic (PLEG): container finished" podID="fef5ac70-c9c9-46ad-aadb-7f5a54264240" containerID="01c84796e70bd9ec6e6a4d39f1a56c3634762419b4084c4661ee2ce538c0b499" exitCode=0 Nov 03 22:52:48.564987 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:48.564531 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx" event={"ID":"fef5ac70-c9c9-46ad-aadb-7f5a54264240","Type":"ContainerDied","Data":"01c84796e70bd9ec6e6a4d39f1a56c3634762419b4084c4661ee2ce538c0b499"} Nov 03 22:52:48.566257 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:48.566232 5084 generic.go:356] "Generic (PLEG): container finished" podID="ed3f9cad-3ed7-440f-b9e3-b3371d649bb8" containerID="5d86de75a304eb49af29e388a84aa327bc42627394297f07aedcdbd5ddb67533" exitCode=0 Nov 03 22:52:48.566387 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:48.566318 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k" event={"ID":"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8","Type":"ContainerDied","Data":"5d86de75a304eb49af29e388a84aa327bc42627394297f07aedcdbd5ddb67533"} Nov 03 22:52:49.743805 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.743778 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k" Nov 03 22:52:49.747229 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.747203 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx" Nov 03 22:52:49.811978 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.811937 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-maas-api-xks-ca\") pod \"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8\" (UID: \"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8\") " Nov 03 22:52:49.812226 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.812019 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-kube-api-access-6ls9k\" (UniqueName: \"kubernetes.io/projected/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-kube-api-access-6ls9k\") pod \"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8\" (UID: \"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8\") " Nov 03 22:52:49.812226 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.812129 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-maas-api-openshift-ca\") pod \"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8\" (UID: \"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8\") " Nov 03 22:52:49.812226 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.812178 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fef5ac70-c9c9-46ad-aadb-7f5a54264240-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef5ac70-c9c9-46ad-aadb-7f5a54264240-maas-api-xks-ca\") pod \"fef5ac70-c9c9-46ad-aadb-7f5a54264240\" (UID: \"fef5ac70-c9c9-46ad-aadb-7f5a54264240\") " Nov 03 22:52:49.812389 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.812239 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef5ac70-c9c9-46ad-aadb-7f5a54264240-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef5ac70-c9c9-46ad-aadb-7f5a54264240-maas-api-openshift-ca\") pod \"fef5ac70-c9c9-46ad-aadb-7f5a54264240\" (UID: \"fef5ac70-c9c9-46ad-aadb-7f5a54264240\") " Nov 03 22:52:49.812389 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.812299 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef5ac70-c9c9-46ad-aadb-7f5a54264240-kube-api-access-2zmx5\" (UniqueName: \"kubernetes.io/projected/fef5ac70-c9c9-46ad-aadb-7f5a54264240-kube-api-access-2zmx5\") pod \"fef5ac70-c9c9-46ad-aadb-7f5a54264240\" (UID: \"fef5ac70-c9c9-46ad-aadb-7f5a54264240\") " Nov 03 22:52:49.814009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.813904 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef5ac70-c9c9-46ad-aadb-7f5a54264240-maas-api-openshift-ca" pod "fef5ac70-c9c9-46ad-aadb-7f5a54264240" (UID: "fef5ac70-c9c9-46ad-aadb-7f5a54264240"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:49.814009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.813959 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-maas-api-openshift-ca" pod "ed3f9cad-3ed7-440f-b9e3-b3371d649bb8" (UID: "ed3f9cad-3ed7-440f-b9e3-b3371d649bb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:49.816476 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.816441 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fef5ac70-c9c9-46ad-aadb-7f5a54264240-maas-api-xks-ca" pod "fef5ac70-c9c9-46ad-aadb-7f5a54264240" (UID: "fef5ac70-c9c9-46ad-aadb-7f5a54264240"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:49.816628 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.816489 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef5ac70-c9c9-46ad-aadb-7f5a54264240-kube-api-access-2zmx5" pod "fef5ac70-c9c9-46ad-aadb-7f5a54264240" (UID: "fef5ac70-c9c9-46ad-aadb-7f5a54264240"). InnerVolumeSpecName "kube-api-access-2zmx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:49.816925 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.816891 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-maas-api-xks-ca" pod "ed3f9cad-3ed7-440f-b9e3-b3371d649bb8" (UID: "ed3f9cad-3ed7-440f-b9e3-b3371d649bb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:49.817195 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.817047 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-kube-api-access-6ls9k" pod "ed3f9cad-3ed7-440f-b9e3-b3371d649bb8" (UID: "ed3f9cad-3ed7-440f-b9e3-b3371d649bb8"). InnerVolumeSpecName "kube-api-access-6ls9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:49.913683 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.913639 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zmx5\" (UniqueName: \"kubernetes.io/projected/fef5ac70-c9c9-46ad-aadb-7f5a54264240-kube-api-access-2zmx5\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:49.913683 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.913678 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-maas-api-xks-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:49.913683 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.913689 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ls9k\" (UniqueName: \"kubernetes.io/projected/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-kube-api-access-6ls9k\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:49.913961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.913698 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8-maas-api-openshift-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:49.913961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.913708 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fef5ac70-c9c9-46ad-aadb-7f5a54264240-maas-api-xks-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:49.913961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:49.913717 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fef5ac70-c9c9-46ad-aadb-7f5a54264240-maas-api-openshift-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:52:50.580909 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:50.580875 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx" Nov 03 22:52:50.581142 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:50.580898 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx" event={"ID":"fef5ac70-c9c9-46ad-aadb-7f5a54264240","Type":"ContainerDied","Data":"238db2634caf753da168a9283d331e60b4c11d01f327c2be5639c42b9b28ae21"} Nov 03 22:52:50.581142 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:50.580953 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="238db2634caf753da168a9283d331e60b4c11d01f327c2be5639c42b9b28ae21" Nov 03 22:52:50.582560 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:50.582533 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k" Nov 03 22:52:50.582560 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:50.582547 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k" event={"ID":"ed3f9cad-3ed7-440f-b9e3-b3371d649bb8","Type":"ContainerDied","Data":"e1ee7575c3ef1d689ff043a0065e3292da81d0c7d309b084fb1ff5d87cbb1f5e"} Nov 03 22:52:50.582775 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:50.582587 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1ee7575c3ef1d689ff043a0065e3292da81d0c7d309b084fb1ff5d87cbb1f5e" Nov 03 22:52:54.593909 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:54.593819 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s"] Nov 03 22:52:54.702407 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:54.702366 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26"] Nov 03 22:52:57.466128 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:57.466082 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm"] Nov 03 22:52:57.466622 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:57.466337 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" podUID="9cd2436f-0b88-411c-bf92-91b6305c85fd" containerName="istio-proxy" containerID="cri-o://f96bbdd790944113a922c0f141e9f8337d9efb476d558da0c4671875f8a2afed" gracePeriod=30 Nov 03 22:52:59.594565 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:59.594510 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s" podUID="2778fdc5-5b89-4783-856c-4e26b7fe30f6" containerName="payload-processing" containerID="cri-o://18b75e2a87c801836d773d3293e3f817efd885932fcfddc0effca79777104681" gracePeriod=25 Nov 03 22:52:59.703345 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:59.703260 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26" podUID="6a069ee1-c643-4a4a-8632-6e02db66d490" containerName="payload-pre-processing" containerID="cri-o://a9667bd04468a9cd0a8804a35170e0efd8def1bbbcb2800cff2d572d5bef5b82" gracePeriod=25 Nov 03 22:52:59.877537 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:59.877408 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s" Nov 03 22:52:59.976954 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:52:59.976918 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26" Nov 03 22:53:00.026817 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.026771 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2778fdc5-5b89-4783-856c-4e26b7fe30f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2778fdc5-5b89-4783-856c-4e26b7fe30f6-plugins-config-volume\") pod \"2778fdc5-5b89-4783-856c-4e26b7fe30f6\" (UID: \"2778fdc5-5b89-4783-856c-4e26b7fe30f6\") " Nov 03 22:53:00.027020 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.026978 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2778fdc5-5b89-4783-856c-4e26b7fe30f6-kube-api-access-hjqmx\" (UniqueName: \"kubernetes.io/projected/2778fdc5-5b89-4783-856c-4e26b7fe30f6-kube-api-access-hjqmx\") pod \"2778fdc5-5b89-4783-856c-4e26b7fe30f6\" (UID: \"2778fdc5-5b89-4783-856c-4e26b7fe30f6\") " Nov 03 22:53:00.027272 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.027241 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2778fdc5-5b89-4783-856c-4e26b7fe30f6-plugins-config-volume" pod "2778fdc5-5b89-4783-856c-4e26b7fe30f6" (UID: "2778fdc5-5b89-4783-856c-4e26b7fe30f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:00.027500 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.027477 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2778fdc5-5b89-4783-856c-4e26b7fe30f6-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:00.029739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.029710 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2778fdc5-5b89-4783-856c-4e26b7fe30f6-kube-api-access-hjqmx" pod "2778fdc5-5b89-4783-856c-4e26b7fe30f6" (UID: "2778fdc5-5b89-4783-856c-4e26b7fe30f6"). InnerVolumeSpecName "kube-api-access-hjqmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:00.128552 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.128439 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a069ee1-c643-4a4a-8632-6e02db66d490-kube-api-access-xr5st\" (UniqueName: \"kubernetes.io/projected/6a069ee1-c643-4a4a-8632-6e02db66d490-kube-api-access-xr5st\") pod \"6a069ee1-c643-4a4a-8632-6e02db66d490\" (UID: \"6a069ee1-c643-4a4a-8632-6e02db66d490\") " Nov 03 22:53:00.128729 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.128643 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a069ee1-c643-4a4a-8632-6e02db66d490-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a069ee1-c643-4a4a-8632-6e02db66d490-plugins-config-volume\") pod \"6a069ee1-c643-4a4a-8632-6e02db66d490\" (UID: \"6a069ee1-c643-4a4a-8632-6e02db66d490\") " Nov 03 22:53:00.128950 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.128932 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjqmx\" (UniqueName: \"kubernetes.io/projected/2778fdc5-5b89-4783-856c-4e26b7fe30f6-kube-api-access-hjqmx\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:00.129152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.129122 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a069ee1-c643-4a4a-8632-6e02db66d490-plugins-config-volume" pod "6a069ee1-c643-4a4a-8632-6e02db66d490" (UID: "6a069ee1-c643-4a4a-8632-6e02db66d490"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:00.130843 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.130811 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a069ee1-c643-4a4a-8632-6e02db66d490-kube-api-access-xr5st" pod "6a069ee1-c643-4a4a-8632-6e02db66d490" (UID: "6a069ee1-c643-4a4a-8632-6e02db66d490"). InnerVolumeSpecName "kube-api-access-xr5st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:00.230176 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.230132 5084 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a069ee1-c643-4a4a-8632-6e02db66d490-plugins-config-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:00.230176 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.230167 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr5st\" (UniqueName: \"kubernetes.io/projected/6a069ee1-c643-4a4a-8632-6e02db66d490-kube-api-access-xr5st\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:00.652591 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.652554 5084 generic.go:356] "Generic (PLEG): container finished" podID="2778fdc5-5b89-4783-856c-4e26b7fe30f6" containerID="18b75e2a87c801836d773d3293e3f817efd885932fcfddc0effca79777104681" exitCode=0 Nov 03 22:53:00.653130 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.652628 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s" Nov 03 22:53:00.653130 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.652650 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s" event={"ID":"2778fdc5-5b89-4783-856c-4e26b7fe30f6","Type":"ContainerDied","Data":"18b75e2a87c801836d773d3293e3f817efd885932fcfddc0effca79777104681"} Nov 03 22:53:00.653130 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.652718 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s" event={"ID":"2778fdc5-5b89-4783-856c-4e26b7fe30f6","Type":"ContainerDied","Data":"35734b03d55b9c84912121a75e8dd479b9fc0741c5e2f35efa98ef386dfe6ca1"} Nov 03 22:53:00.653130 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.652742 5084 scope.go:122] "RemoveContainer" containerID="18b75e2a87c801836d773d3293e3f817efd885932fcfddc0effca79777104681" Nov 03 22:53:00.654494 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.654372 5084 generic.go:356] "Generic (PLEG): container finished" podID="6a069ee1-c643-4a4a-8632-6e02db66d490" containerID="a9667bd04468a9cd0a8804a35170e0efd8def1bbbcb2800cff2d572d5bef5b82" exitCode=0 Nov 03 22:53:00.654494 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.654453 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26" Nov 03 22:53:00.654608 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.654452 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26" event={"ID":"6a069ee1-c643-4a4a-8632-6e02db66d490","Type":"ContainerDied","Data":"a9667bd04468a9cd0a8804a35170e0efd8def1bbbcb2800cff2d572d5bef5b82"} Nov 03 22:53:00.654608 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.654578 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26" event={"ID":"6a069ee1-c643-4a4a-8632-6e02db66d490","Type":"ContainerDied","Data":"748e060ef6e0acf66727cc75e0f2d3f1bac67996981e436b7b75de836e541545"} Nov 03 22:53:00.664408 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.664379 5084 scope.go:122] "RemoveContainer" containerID="18b75e2a87c801836d773d3293e3f817efd885932fcfddc0effca79777104681" Nov 03 22:53:00.664743 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:53:00.664722 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b75e2a87c801836d773d3293e3f817efd885932fcfddc0effca79777104681\": container with ID starting with 18b75e2a87c801836d773d3293e3f817efd885932fcfddc0effca79777104681 not found: ID does not exist" containerID="18b75e2a87c801836d773d3293e3f817efd885932fcfddc0effca79777104681" Nov 03 22:53:00.664806 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.664754 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b75e2a87c801836d773d3293e3f817efd885932fcfddc0effca79777104681"} err="failed to get container status \"18b75e2a87c801836d773d3293e3f817efd885932fcfddc0effca79777104681\": rpc error: code = NotFound desc = could not find container \"18b75e2a87c801836d773d3293e3f817efd885932fcfddc0effca79777104681\": container with ID starting with 18b75e2a87c801836d773d3293e3f817efd885932fcfddc0effca79777104681 not found: ID does not exist" Nov 03 22:53:00.664806 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.664775 5084 scope.go:122] "RemoveContainer" containerID="a9667bd04468a9cd0a8804a35170e0efd8def1bbbcb2800cff2d572d5bef5b82" Nov 03 22:53:00.674995 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.674964 5084 scope.go:122] "RemoveContainer" containerID="a9667bd04468a9cd0a8804a35170e0efd8def1bbbcb2800cff2d572d5bef5b82" Nov 03 22:53:00.675354 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:53:00.675331 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9667bd04468a9cd0a8804a35170e0efd8def1bbbcb2800cff2d572d5bef5b82\": container with ID starting with a9667bd04468a9cd0a8804a35170e0efd8def1bbbcb2800cff2d572d5bef5b82 not found: ID does not exist" containerID="a9667bd04468a9cd0a8804a35170e0efd8def1bbbcb2800cff2d572d5bef5b82" Nov 03 22:53:00.675421 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.675364 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9667bd04468a9cd0a8804a35170e0efd8def1bbbcb2800cff2d572d5bef5b82"} err="failed to get container status \"a9667bd04468a9cd0a8804a35170e0efd8def1bbbcb2800cff2d572d5bef5b82\": rpc error: code = NotFound desc = could not find container \"a9667bd04468a9cd0a8804a35170e0efd8def1bbbcb2800cff2d572d5bef5b82\": container with ID starting with a9667bd04468a9cd0a8804a35170e0efd8def1bbbcb2800cff2d572d5bef5b82 not found: ID does not exist" Nov 03 22:53:00.821535 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.821495 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26"] Nov 03 22:53:00.825083 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.825036 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-2a6801-77cfd9b767-2ns26"] Nov 03 22:53:00.936420 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.936323 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s"] Nov 03 22:53:00.942428 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:00.942388 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-2a6801-7cfb77d64-w9n9s"] Nov 03 22:53:02.628739 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.628705 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2778fdc5-5b89-4783-856c-4e26b7fe30f6" path="/var/lib/kubelet/pods/2778fdc5-5b89-4783-856c-4e26b7fe30f6/volumes" Nov 03 22:53:02.629221 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.629202 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a069ee1-c643-4a4a-8632-6e02db66d490" path="/var/lib/kubelet/pods/6a069ee1-c643-4a4a-8632-6e02db66d490/volumes" Nov 03 22:53:02.674414 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.674373 5084 generic.go:356] "Generic (PLEG): container finished" podID="9cd2436f-0b88-411c-bf92-91b6305c85fd" containerID="f96bbdd790944113a922c0f141e9f8337d9efb476d558da0c4671875f8a2afed" exitCode=0 Nov 03 22:53:02.674414 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.674400 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" event={"ID":"9cd2436f-0b88-411c-bf92-91b6305c85fd","Type":"ContainerDied","Data":"f96bbdd790944113a922c0f141e9f8337d9efb476d558da0c4671875f8a2afed"} Nov 03 22:53:02.747206 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.747114 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:53:02.858723 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.858685 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-workload-certs\") pod \"9cd2436f-0b88-411c-bf92-91b6305c85fd\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " Nov 03 22:53:02.858950 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.858741 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-workload-socket\") pod \"9cd2436f-0b88-411c-bf92-91b6305c85fd\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " Nov 03 22:53:02.858950 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.858768 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-envoy\") pod \"9cd2436f-0b88-411c-bf92-91b6305c85fd\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " Nov 03 22:53:02.858950 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.858791 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cd2436f-0b88-411c-bf92-91b6305c85fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd2436f-0b88-411c-bf92-91b6305c85fd-istiod-ca-cert\") pod \"9cd2436f-0b88-411c-bf92-91b6305c85fd\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " Nov 03 22:53:02.858950 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.858835 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd2436f-0b88-411c-bf92-91b6305c85fd-kube-api-access-8wgnf\" (UniqueName: \"kubernetes.io/projected/9cd2436f-0b88-411c-bf92-91b6305c85fd-kube-api-access-8wgnf\") pod \"9cd2436f-0b88-411c-bf92-91b6305c85fd\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " Nov 03 22:53:02.858950 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.858883 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-token\" (UniqueName: \"kubernetes.io/projected/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-token\") pod \"9cd2436f-0b88-411c-bf92-91b6305c85fd\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " Nov 03 22:53:02.858950 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.858945 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-data\") pod \"9cd2436f-0b88-411c-bf92-91b6305c85fd\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " Nov 03 22:53:02.859310 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.858981 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-credential-socket\") pod \"9cd2436f-0b88-411c-bf92-91b6305c85fd\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " Nov 03 22:53:02.859310 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.859027 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-workload-certs" pod "9cd2436f-0b88-411c-bf92-91b6305c85fd" (UID: "9cd2436f-0b88-411c-bf92-91b6305c85fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.859310 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.859136 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-podinfo\") pod \"9cd2436f-0b88-411c-bf92-91b6305c85fd\" (UID: \"9cd2436f-0b88-411c-bf92-91b6305c85fd\") " Nov 03 22:53:02.859310 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.859213 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-workload-socket" pod "9cd2436f-0b88-411c-bf92-91b6305c85fd" (UID: "9cd2436f-0b88-411c-bf92-91b6305c85fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.859310 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.859234 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-data" pod "9cd2436f-0b88-411c-bf92-91b6305c85fd" (UID: "9cd2436f-0b88-411c-bf92-91b6305c85fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.859525 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.859348 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-credential-socket" pod "9cd2436f-0b88-411c-bf92-91b6305c85fd" (UID: "9cd2436f-0b88-411c-bf92-91b6305c85fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.859582 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.859564 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd2436f-0b88-411c-bf92-91b6305c85fd-istiod-ca-cert" pod "9cd2436f-0b88-411c-bf92-91b6305c85fd" (UID: "9cd2436f-0b88-411c-bf92-91b6305c85fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:02.859652 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.859628 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.859706 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.859667 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.859706 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.859682 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.859706 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.859696 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.861599 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.861562 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-envoy" pod "9cd2436f-0b88-411c-bf92-91b6305c85fd" (UID: "9cd2436f-0b88-411c-bf92-91b6305c85fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:02.861599 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.861573 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-podinfo" pod "9cd2436f-0b88-411c-bf92-91b6305c85fd" (UID: "9cd2436f-0b88-411c-bf92-91b6305c85fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:53:02.861788 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.861622 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd2436f-0b88-411c-bf92-91b6305c85fd-kube-api-access-8wgnf" pod "9cd2436f-0b88-411c-bf92-91b6305c85fd" (UID: "9cd2436f-0b88-411c-bf92-91b6305c85fd"). InnerVolumeSpecName "kube-api-access-8wgnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:02.862009 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.861981 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-token" pod "9cd2436f-0b88-411c-bf92-91b6305c85fd" (UID: "9cd2436f-0b88-411c-bf92-91b6305c85fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:02.960614 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.960570 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.960614 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.960613 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.960856 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.960631 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd2436f-0b88-411c-bf92-91b6305c85fd-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.960856 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.960644 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wgnf\" (UniqueName: \"kubernetes.io/projected/9cd2436f-0b88-411c-bf92-91b6305c85fd-kube-api-access-8wgnf\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:02.960856 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:02.960661 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd2436f-0b88-411c-bf92-91b6305c85fd-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:03.684725 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:03.684689 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" Nov 03 22:53:03.684725 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:03.684706 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm" event={"ID":"9cd2436f-0b88-411c-bf92-91b6305c85fd","Type":"ContainerDied","Data":"19e27e65d900e6a9b26db350b1ad3ae96d422b2e4ee1ebedfaa105bf78961797"} Nov 03 22:53:03.685279 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:03.684776 5084 scope.go:122] "RemoveContainer" containerID="f96bbdd790944113a922c0f141e9f8337d9efb476d558da0c4671875f8a2afed" Nov 03 22:53:03.904247 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:03.904201 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm"] Nov 03 22:53:03.955006 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:03.954911 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2a6801-openshift-default-78f85489f9-rl6wm"] Nov 03 22:53:04.633658 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:04.633609 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd2436f-0b88-411c-bf92-91b6305c85fd" path="/var/lib/kubelet/pods/9cd2436f-0b88-411c-bf92-91b6305c85fd/volumes" Nov 03 22:53:08.592099 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:08.592021 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46"] Nov 03 22:53:08.592559 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:08.592262 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" podUID="060aaaff-87eb-4800-99af-122329708c89" containerName="maas-api" containerID="cri-o://015995cb6ce852319894c31d0a94c4f7afde0facbb09bbedcd19cd5fc10561e4" gracePeriod=30 Nov 03 22:53:08.720297 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:08.720260 5084 generic.go:356] "Generic (PLEG): container finished" podID="060aaaff-87eb-4800-99af-122329708c89" containerID="015995cb6ce852319894c31d0a94c4f7afde0facbb09bbedcd19cd5fc10561e4" exitCode=0 Nov 03 22:53:08.720503 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:08.720449 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" event={"ID":"060aaaff-87eb-4800-99af-122329708c89","Type":"ContainerDied","Data":"015995cb6ce852319894c31d0a94c4f7afde0facbb09bbedcd19cd5fc10561e4"} Nov 03 22:53:08.895635 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:08.895603 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:53:09.026465 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:09.026422 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/060aaaff-87eb-4800-99af-122329708c89-metrics-certs\" (UniqueName: \"kubernetes.io/secret/060aaaff-87eb-4800-99af-122329708c89-metrics-certs\") pod \"060aaaff-87eb-4800-99af-122329708c89\" (UID: \"060aaaff-87eb-4800-99af-122329708c89\") " Nov 03 22:53:09.026668 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:09.026522 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/060aaaff-87eb-4800-99af-122329708c89-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/060aaaff-87eb-4800-99af-122329708c89-maas-api-tls\") pod \"060aaaff-87eb-4800-99af-122329708c89\" (UID: \"060aaaff-87eb-4800-99af-122329708c89\") " Nov 03 22:53:09.026668 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:09.026573 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060aaaff-87eb-4800-99af-122329708c89-kube-api-access-9lxzc\" (UniqueName: \"kubernetes.io/projected/060aaaff-87eb-4800-99af-122329708c89-kube-api-access-9lxzc\") pod \"060aaaff-87eb-4800-99af-122329708c89\" (UID: \"060aaaff-87eb-4800-99af-122329708c89\") " Nov 03 22:53:09.026668 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:09.026656 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060aaaff-87eb-4800-99af-122329708c89-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/060aaaff-87eb-4800-99af-122329708c89-openshift-service-ca\") pod \"060aaaff-87eb-4800-99af-122329708c89\" (UID: \"060aaaff-87eb-4800-99af-122329708c89\") " Nov 03 22:53:09.027107 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:09.027076 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060aaaff-87eb-4800-99af-122329708c89-openshift-service-ca" pod "060aaaff-87eb-4800-99af-122329708c89" (UID: "060aaaff-87eb-4800-99af-122329708c89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:09.029087 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:09.029041 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060aaaff-87eb-4800-99af-122329708c89-metrics-certs" pod "060aaaff-87eb-4800-99af-122329708c89" (UID: "060aaaff-87eb-4800-99af-122329708c89"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:09.029290 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:09.029267 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060aaaff-87eb-4800-99af-122329708c89-maas-api-tls" pod "060aaaff-87eb-4800-99af-122329708c89" (UID: "060aaaff-87eb-4800-99af-122329708c89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:09.029374 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:09.029348 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060aaaff-87eb-4800-99af-122329708c89-kube-api-access-9lxzc" pod "060aaaff-87eb-4800-99af-122329708c89" (UID: "060aaaff-87eb-4800-99af-122329708c89"). InnerVolumeSpecName "kube-api-access-9lxzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:09.136191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:09.133762 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/060aaaff-87eb-4800-99af-122329708c89-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:09.136191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:09.133818 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/060aaaff-87eb-4800-99af-122329708c89-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:09.136191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:09.133831 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/060aaaff-87eb-4800-99af-122329708c89-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:09.136191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:09.133847 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lxzc\" (UniqueName: \"kubernetes.io/projected/060aaaff-87eb-4800-99af-122329708c89-kube-api-access-9lxzc\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:09.728982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:09.728949 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" Nov 03 22:53:09.728982 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:09.728962 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46" event={"ID":"060aaaff-87eb-4800-99af-122329708c89","Type":"ContainerDied","Data":"f33d5fa5172b57bad439071375dd65b82e667bd9bd4f33cf634e0ba42cf115e6"} Nov 03 22:53:09.729576 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:09.729029 5084 scope.go:122] "RemoveContainer" containerID="015995cb6ce852319894c31d0a94c4f7afde0facbb09bbedcd19cd5fc10561e4" Nov 03 22:53:09.898816 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:09.898778 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46"] Nov 03 22:53:09.949560 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:09.949519 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-2a6801-5d7997b98d-vxs46"] Nov 03 22:53:10.628013 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:10.627972 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060aaaff-87eb-4800-99af-122329708c89" path="/var/lib/kubelet/pods/060aaaff-87eb-4800-99af-122329708c89/volumes" Nov 03 22:53:12.024274 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:53:12.024225 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:53:16.263240 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:16.263200 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb"] Nov 03 22:53:16.265765 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:16.263425 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" podUID="5ca0b679-54b7-44fe-8a91-94de18621666" containerName="istio-proxy" containerID="cri-o://0d9b92a8c9a0d02f96acb1a5695a01f684ebc850f6359f29760f9fab3e1e70c3" gracePeriod=30 Nov 03 22:53:16.997630 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:16.997595 5084 scope.go:122] "RemoveContainer" containerID="9726ea99f773993eedec23e63909f6fef54878b04ee5ef05bda9f46a8d8bb7e2" Nov 03 22:53:21.549692 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.549618 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:53:21.664252 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.664208 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-istio-data\") pod \"5ca0b679-54b7-44fe-8a91-94de18621666\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " Nov 03 22:53:21.664468 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.664267 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca0b679-54b7-44fe-8a91-94de18621666-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca0b679-54b7-44fe-8a91-94de18621666-istiod-ca-cert\") pod \"5ca0b679-54b7-44fe-8a91-94de18621666\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " Nov 03 22:53:21.664468 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.664290 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca0b679-54b7-44fe-8a91-94de18621666-istio-token\" (UniqueName: \"kubernetes.io/projected/5ca0b679-54b7-44fe-8a91-94de18621666-istio-token\") pod \"5ca0b679-54b7-44fe-8a91-94de18621666\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " Nov 03 22:53:21.664468 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.664353 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-credential-socket\") pod \"5ca0b679-54b7-44fe-8a91-94de18621666\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " Nov 03 22:53:21.664468 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.664386 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-istio-envoy\") pod \"5ca0b679-54b7-44fe-8a91-94de18621666\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " Nov 03 22:53:21.664468 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.664418 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-workload-socket\") pod \"5ca0b679-54b7-44fe-8a91-94de18621666\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " Nov 03 22:53:21.664468 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.664445 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-workload-certs\") pod \"5ca0b679-54b7-44fe-8a91-94de18621666\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " Nov 03 22:53:21.664817 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.664490 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ca0b679-54b7-44fe-8a91-94de18621666-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca0b679-54b7-44fe-8a91-94de18621666-istio-podinfo\") pod \"5ca0b679-54b7-44fe-8a91-94de18621666\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " Nov 03 22:53:21.664817 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.664558 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-istio-data" pod "5ca0b679-54b7-44fe-8a91-94de18621666" (UID: "5ca0b679-54b7-44fe-8a91-94de18621666"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:21.664817 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.664578 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca0b679-54b7-44fe-8a91-94de18621666-kube-api-access-lbgt5\" (UniqueName: \"kubernetes.io/projected/5ca0b679-54b7-44fe-8a91-94de18621666-kube-api-access-lbgt5\") pod \"5ca0b679-54b7-44fe-8a91-94de18621666\" (UID: \"5ca0b679-54b7-44fe-8a91-94de18621666\") " Nov 03 22:53:21.665034 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.664820 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-credential-socket" pod "5ca0b679-54b7-44fe-8a91-94de18621666" (UID: "5ca0b679-54b7-44fe-8a91-94de18621666"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:21.665034 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.664830 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-workload-socket" pod "5ca0b679-54b7-44fe-8a91-94de18621666" (UID: "5ca0b679-54b7-44fe-8a91-94de18621666"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:21.665034 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.664850 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca0b679-54b7-44fe-8a91-94de18621666-istiod-ca-cert" pod "5ca0b679-54b7-44fe-8a91-94de18621666" (UID: "5ca0b679-54b7-44fe-8a91-94de18621666"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:21.667541 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.667490 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca0b679-54b7-44fe-8a91-94de18621666-istio-token" pod "5ca0b679-54b7-44fe-8a91-94de18621666" (UID: "5ca0b679-54b7-44fe-8a91-94de18621666"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:21.667698 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.667585 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-istio-envoy" pod "5ca0b679-54b7-44fe-8a91-94de18621666" (UID: "5ca0b679-54b7-44fe-8a91-94de18621666"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:21.667698 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.665612 5084 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-istio-data\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:21.667698 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.667628 5084 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca0b679-54b7-44fe-8a91-94de18621666-istiod-ca-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:21.667698 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.667655 5084 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-credential-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:21.667698 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.667671 5084 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-workload-socket\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:21.667961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.667729 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ca0b679-54b7-44fe-8a91-94de18621666-istio-podinfo" pod "5ca0b679-54b7-44fe-8a91-94de18621666" (UID: "5ca0b679-54b7-44fe-8a91-94de18621666"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:53:21.670120 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.668213 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-workload-certs" pod "5ca0b679-54b7-44fe-8a91-94de18621666" (UID: "5ca0b679-54b7-44fe-8a91-94de18621666"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:21.676961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.676900 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca0b679-54b7-44fe-8a91-94de18621666-kube-api-access-lbgt5" pod "5ca0b679-54b7-44fe-8a91-94de18621666" (UID: "5ca0b679-54b7-44fe-8a91-94de18621666"). InnerVolumeSpecName "kube-api-access-lbgt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:21.768484 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.768448 5084 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-istio-envoy\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:21.768484 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.768481 5084 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca0b679-54b7-44fe-8a91-94de18621666-workload-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:21.768484 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.768492 5084 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca0b679-54b7-44fe-8a91-94de18621666-istio-podinfo\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:21.768733 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.768503 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbgt5\" (UniqueName: \"kubernetes.io/projected/5ca0b679-54b7-44fe-8a91-94de18621666-kube-api-access-lbgt5\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:21.768733 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.768513 5084 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca0b679-54b7-44fe-8a91-94de18621666-istio-token\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:21.806299 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.806195 5084 generic.go:356] "Generic (PLEG): container finished" podID="5ca0b679-54b7-44fe-8a91-94de18621666" containerID="0d9b92a8c9a0d02f96acb1a5695a01f684ebc850f6359f29760f9fab3e1e70c3" exitCode=0 Nov 03 22:53:21.806299 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.806266 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" Nov 03 22:53:21.806549 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.806285 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" event={"ID":"5ca0b679-54b7-44fe-8a91-94de18621666","Type":"ContainerDied","Data":"0d9b92a8c9a0d02f96acb1a5695a01f684ebc850f6359f29760f9fab3e1e70c3"} Nov 03 22:53:21.806549 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.806381 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb" event={"ID":"5ca0b679-54b7-44fe-8a91-94de18621666","Type":"ContainerDied","Data":"ed21ab1bdf35f1057e6212fca359d39e8e17d105631d0ebdcee407b2cb30df7f"} Nov 03 22:53:21.806549 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.806412 5084 scope.go:122] "RemoveContainer" containerID="0d9b92a8c9a0d02f96acb1a5695a01f684ebc850f6359f29760f9fab3e1e70c3" Nov 03 22:53:21.817476 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.817450 5084 scope.go:122] "RemoveContainer" containerID="0d9b92a8c9a0d02f96acb1a5695a01f684ebc850f6359f29760f9fab3e1e70c3" Nov 03 22:53:21.817826 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:53:21.817780 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9b92a8c9a0d02f96acb1a5695a01f684ebc850f6359f29760f9fab3e1e70c3\": container with ID starting with 0d9b92a8c9a0d02f96acb1a5695a01f684ebc850f6359f29760f9fab3e1e70c3 not found: ID does not exist" containerID="0d9b92a8c9a0d02f96acb1a5695a01f684ebc850f6359f29760f9fab3e1e70c3" Nov 03 22:53:21.817907 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.817836 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9b92a8c9a0d02f96acb1a5695a01f684ebc850f6359f29760f9fab3e1e70c3"} err="failed to get container status \"0d9b92a8c9a0d02f96acb1a5695a01f684ebc850f6359f29760f9fab3e1e70c3\": rpc error: code = NotFound desc = could not find container \"0d9b92a8c9a0d02f96acb1a5695a01f684ebc850f6359f29760f9fab3e1e70c3\": container with ID starting with 0d9b92a8c9a0d02f96acb1a5695a01f684ebc850f6359f29760f9fab3e1e70c3 not found: ID does not exist" Nov 03 22:53:21.975357 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.975318 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb"] Nov 03 22:53:21.978361 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:21.978322 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-906673-openshift-default-78f9f69c67-ggpjb"] Nov 03 22:53:22.627378 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:22.627336 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ca0b679-54b7-44fe-8a91-94de18621666" path="/var/lib/kubelet/pods/5ca0b679-54b7-44fe-8a91-94de18621666/volumes" Nov 03 22:53:35.533473 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.533427 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26"] Nov 03 22:53:35.533897 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.533883 5084 cpu_manager.go:414] "containerMap: removing container" podUID="9cd2436f-0b88-411c-bf92-91b6305c85fd" containerName="istio-proxy" Nov 03 22:53:35.533897 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.533896 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd2436f-0b88-411c-bf92-91b6305c85fd" containerName="istio-proxy" Nov 03 22:53:35.533978 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.533921 5084 cpu_manager.go:414] "containerMap: removing container" podUID="060aaaff-87eb-4800-99af-122329708c89" containerName="maas-api" Nov 03 22:53:35.533978 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.533926 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060aaaff-87eb-4800-99af-122329708c89" containerName="maas-api" Nov 03 22:53:35.533978 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.533940 5084 cpu_manager.go:414] "containerMap: removing container" podUID="ed3f9cad-3ed7-440f-b9e3-b3371d649bb8" containerName="revoke-keys" Nov 03 22:53:35.533978 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.533945 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3f9cad-3ed7-440f-b9e3-b3371d649bb8" containerName="revoke-keys" Nov 03 22:53:35.533978 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.533952 5084 cpu_manager.go:414] "containerMap: removing container" podUID="6a069ee1-c643-4a4a-8632-6e02db66d490" containerName="payload-pre-processing" Nov 03 22:53:35.533978 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.533956 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a069ee1-c643-4a4a-8632-6e02db66d490" containerName="payload-pre-processing" Nov 03 22:53:35.533978 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.533966 5084 cpu_manager.go:414] "containerMap: removing container" podUID="fef5ac70-c9c9-46ad-aadb-7f5a54264240" containerName="revoke-keys" Nov 03 22:53:35.533978 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.533971 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef5ac70-c9c9-46ad-aadb-7f5a54264240" containerName="revoke-keys" Nov 03 22:53:35.533978 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.533979 5084 cpu_manager.go:414] "containerMap: removing container" podUID="2778fdc5-5b89-4783-856c-4e26b7fe30f6" containerName="payload-processing" Nov 03 22:53:35.534294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.533983 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2778fdc5-5b89-4783-856c-4e26b7fe30f6" containerName="payload-processing" Nov 03 22:53:35.534294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.533994 5084 cpu_manager.go:414] "containerMap: removing container" podUID="5ca0b679-54b7-44fe-8a91-94de18621666" containerName="istio-proxy" Nov 03 22:53:35.534294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.533999 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca0b679-54b7-44fe-8a91-94de18621666" containerName="istio-proxy" Nov 03 22:53:35.534294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.534078 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd2436f-0b88-411c-bf92-91b6305c85fd" containerName="istio-proxy" Nov 03 22:53:35.534294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.534094 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="060aaaff-87eb-4800-99af-122329708c89" containerName="maas-api" Nov 03 22:53:35.534294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.534104 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3f9cad-3ed7-440f-b9e3-b3371d649bb8" containerName="revoke-keys" Nov 03 22:53:35.534294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.534113 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a069ee1-c643-4a4a-8632-6e02db66d490" containerName="payload-pre-processing" Nov 03 22:53:35.534294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.534122 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef5ac70-c9c9-46ad-aadb-7f5a54264240" containerName="revoke-keys" Nov 03 22:53:35.534294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.534130 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="2778fdc5-5b89-4783-856c-4e26b7fe30f6" containerName="payload-processing" Nov 03 22:53:35.534294 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.534140 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca0b679-54b7-44fe-8a91-94de18621666" containerName="istio-proxy" Nov 03 22:53:35.538773 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.538735 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26" Nov 03 22:53:35.592272 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.592235 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26"] Nov 03 22:53:35.620145 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.620094 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5zm9\" (UniqueName: \"kubernetes.io/projected/416d4945-1652-4bb1-996d-db1e4d2a8edd-kube-api-access-c5zm9\") pod \"maas-api-revoke-sub-ac0d55ff-94l26\" (UID: \"416d4945-1652-4bb1-996d-db1e4d2a8edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26" Nov 03 22:53:35.620341 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.620193 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416d4945-1652-4bb1-996d-db1e4d2a8edd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac0d55ff-94l26\" (UID: \"416d4945-1652-4bb1-996d-db1e4d2a8edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26" Nov 03 22:53:35.620341 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.620242 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416d4945-1652-4bb1-996d-db1e4d2a8edd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac0d55ff-94l26\" (UID: \"416d4945-1652-4bb1-996d-db1e4d2a8edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26" Nov 03 22:53:35.640363 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.640327 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 22:53:35.640549 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.640332 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:53:35.640549 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.640341 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:53:35.640549 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.640377 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:53:35.721163 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.721122 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416d4945-1652-4bb1-996d-db1e4d2a8edd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac0d55ff-94l26\" (UID: \"416d4945-1652-4bb1-996d-db1e4d2a8edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26" Nov 03 22:53:35.721435 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.721408 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zm9\" (UniqueName: \"kubernetes.io/projected/416d4945-1652-4bb1-996d-db1e4d2a8edd-kube-api-access-c5zm9\") pod \"maas-api-revoke-sub-ac0d55ff-94l26\" (UID: \"416d4945-1652-4bb1-996d-db1e4d2a8edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26" Nov 03 22:53:35.721618 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.721596 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416d4945-1652-4bb1-996d-db1e4d2a8edd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac0d55ff-94l26\" (UID: \"416d4945-1652-4bb1-996d-db1e4d2a8edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26" Nov 03 22:53:35.721985 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.721965 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416d4945-1652-4bb1-996d-db1e4d2a8edd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac0d55ff-94l26\" (UID: \"416d4945-1652-4bb1-996d-db1e4d2a8edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26" Nov 03 22:53:35.723892 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.723870 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416d4945-1652-4bb1-996d-db1e4d2a8edd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac0d55ff-94l26\" (UID: \"416d4945-1652-4bb1-996d-db1e4d2a8edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26" Nov 03 22:53:35.780049 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.780007 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5zm9\" (UniqueName: \"kubernetes.io/projected/416d4945-1652-4bb1-996d-db1e4d2a8edd-kube-api-access-c5zm9\") pod \"maas-api-revoke-sub-ac0d55ff-94l26\" (UID: \"416d4945-1652-4bb1-996d-db1e4d2a8edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26" Nov 03 22:53:35.853942 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:35.853846 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26" Nov 03 22:53:36.234276 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:53:36.234235 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416d4945_1652_4bb1_996d_db1e4d2a8edd.slice/crio-b29d96179d4843324ae4fcd19f0c8cc32435c8a17187b57d0507921b670bc2ed WatchSource:0}: Error finding container b29d96179d4843324ae4fcd19f0c8cc32435c8a17187b57d0507921b670bc2ed: Status 404 returned error can't find the container with id b29d96179d4843324ae4fcd19f0c8cc32435c8a17187b57d0507921b670bc2ed Nov 03 22:53:36.236528 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:36.236494 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26"] Nov 03 22:53:36.911376 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:36.911271 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26" event={"ID":"416d4945-1652-4bb1-996d-db1e4d2a8edd","Type":"ContainerStarted","Data":"3a14533ecbc70bf0cd7f83075c5cc8768e6ddf0754bbbf4206cfa6e921512fd0"} Nov 03 22:53:36.911376 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:36.911343 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26" event={"ID":"416d4945-1652-4bb1-996d-db1e4d2a8edd","Type":"ContainerStarted","Data":"b29d96179d4843324ae4fcd19f0c8cc32435c8a17187b57d0507921b670bc2ed"} Nov 03 22:53:37.029551 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:37.029492 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26" podStartSLOduration=2.029471482 podStartE2EDuration="2.029471482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:37.026660602 +0000 UTC m=+2784.620692576" watchObservedRunningTime="2026-11-03 22:53:37.029471482 +0000 UTC m=+2784.623503456" Nov 03 22:53:37.919196 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:37.919150 5084 generic.go:356] "Generic (PLEG): container finished" podID="416d4945-1652-4bb1-996d-db1e4d2a8edd" containerID="3a14533ecbc70bf0cd7f83075c5cc8768e6ddf0754bbbf4206cfa6e921512fd0" exitCode=0 Nov 03 22:53:37.919612 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:37.919218 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26" event={"ID":"416d4945-1652-4bb1-996d-db1e4d2a8edd","Type":"ContainerDied","Data":"3a14533ecbc70bf0cd7f83075c5cc8768e6ddf0754bbbf4206cfa6e921512fd0"} Nov 03 22:53:39.081152 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:39.081121 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26" Nov 03 22:53:39.259378 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:39.259288 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416d4945-1652-4bb1-996d-db1e4d2a8edd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416d4945-1652-4bb1-996d-db1e4d2a8edd-maas-api-openshift-ca\") pod \"416d4945-1652-4bb1-996d-db1e4d2a8edd\" (UID: \"416d4945-1652-4bb1-996d-db1e4d2a8edd\") " Nov 03 22:53:39.259378 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:39.259334 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416d4945-1652-4bb1-996d-db1e4d2a8edd-kube-api-access-c5zm9\" (UniqueName: \"kubernetes.io/projected/416d4945-1652-4bb1-996d-db1e4d2a8edd-kube-api-access-c5zm9\") pod \"416d4945-1652-4bb1-996d-db1e4d2a8edd\" (UID: \"416d4945-1652-4bb1-996d-db1e4d2a8edd\") " Nov 03 22:53:39.259585 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:39.259525 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/416d4945-1652-4bb1-996d-db1e4d2a8edd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416d4945-1652-4bb1-996d-db1e4d2a8edd-maas-api-xks-ca\") pod \"416d4945-1652-4bb1-996d-db1e4d2a8edd\" (UID: \"416d4945-1652-4bb1-996d-db1e4d2a8edd\") " Nov 03 22:53:39.259763 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:39.259739 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416d4945-1652-4bb1-996d-db1e4d2a8edd-maas-api-openshift-ca" pod "416d4945-1652-4bb1-996d-db1e4d2a8edd" (UID: "416d4945-1652-4bb1-996d-db1e4d2a8edd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:39.259918 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:39.259893 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416d4945-1652-4bb1-996d-db1e4d2a8edd-maas-api-openshift-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:39.261624 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:39.261594 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416d4945-1652-4bb1-996d-db1e4d2a8edd-kube-api-access-c5zm9" pod "416d4945-1652-4bb1-996d-db1e4d2a8edd" (UID: "416d4945-1652-4bb1-996d-db1e4d2a8edd"). InnerVolumeSpecName "kube-api-access-c5zm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:39.261760 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:39.261711 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416d4945-1652-4bb1-996d-db1e4d2a8edd-maas-api-xks-ca" pod "416d4945-1652-4bb1-996d-db1e4d2a8edd" (UID: "416d4945-1652-4bb1-996d-db1e4d2a8edd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:39.361291 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:39.361241 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5zm9\" (UniqueName: \"kubernetes.io/projected/416d4945-1652-4bb1-996d-db1e4d2a8edd-kube-api-access-c5zm9\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:39.361291 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:39.361284 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416d4945-1652-4bb1-996d-db1e4d2a8edd-maas-api-xks-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:53:39.932186 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:39.932152 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26" Nov 03 22:53:39.932186 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:39.932167 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26" event={"ID":"416d4945-1652-4bb1-996d-db1e4d2a8edd","Type":"ContainerDied","Data":"b29d96179d4843324ae4fcd19f0c8cc32435c8a17187b57d0507921b670bc2ed"} Nov 03 22:53:39.932405 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:53:39.932217 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b29d96179d4843324ae4fcd19f0c8cc32435c8a17187b57d0507921b670bc2ed" Nov 03 22:53:42.023447 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:53:42.023406 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:54:05.887195 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:05.887150 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57d7bb69fb-scrzg"] Nov 03 22:54:05.887799 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:05.887404 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" podUID="5bb4c5a2-c6c1-4be9-9656-5eecee18ce49" containerName="manager" containerID="cri-o://80574e6f20cc41d8e49c6261bc33f056fc1520d8431785ba6e6188c690e790a6" gracePeriod=30 Nov 03 22:54:06.097678 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:06.097640 5084 generic.go:356] "Generic (PLEG): container finished" podID="5bb4c5a2-c6c1-4be9-9656-5eecee18ce49" containerID="80574e6f20cc41d8e49c6261bc33f056fc1520d8431785ba6e6188c690e790a6" exitCode=0 Nov 03 22:54:06.097896 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:06.097716 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" event={"ID":"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49","Type":"ContainerDied","Data":"80574e6f20cc41d8e49c6261bc33f056fc1520d8431785ba6e6188c690e790a6"} Nov 03 22:54:06.097896 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:06.097808 5084 scope.go:122] "RemoveContainer" containerID="e87355f8b45c32ef5cffc001590054967f8e05bfdadba0cca1f87ca6cb15d7b1" Nov 03 22:54:06.168811 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:06.168775 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:54:06.253776 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:06.253737 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-webhook-cert\" (UniqueName: \"kubernetes.io/secret/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-webhook-cert\") pod \"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49\" (UID: \"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49\") " Nov 03 22:54:06.254017 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:06.253894 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-metrics-certs\") pod \"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49\" (UID: \"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49\") " Nov 03 22:54:06.254017 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:06.253993 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-kube-api-access-hxbq8\" (UniqueName: \"kubernetes.io/projected/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-kube-api-access-hxbq8\") pod \"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49\" (UID: \"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49\") " Nov 03 22:54:06.256120 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:06.256083 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-webhook-cert" pod "5bb4c5a2-c6c1-4be9-9656-5eecee18ce49" (UID: "5bb4c5a2-c6c1-4be9-9656-5eecee18ce49"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:06.256268 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:06.256122 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-metrics-certs" pod "5bb4c5a2-c6c1-4be9-9656-5eecee18ce49" (UID: "5bb4c5a2-c6c1-4be9-9656-5eecee18ce49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:06.256268 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:06.256125 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-kube-api-access-hxbq8" pod "5bb4c5a2-c6c1-4be9-9656-5eecee18ce49" (UID: "5bb4c5a2-c6c1-4be9-9656-5eecee18ce49"). InnerVolumeSpecName "kube-api-access-hxbq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:06.355317 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:06.355283 5084 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-webhook-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:06.355317 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:06.355313 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:06.355527 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:06.355323 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxbq8\" (UniqueName: \"kubernetes.io/projected/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49-kube-api-access-hxbq8\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:07.104735 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:07.104703 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" Nov 03 22:54:07.105217 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:07.104738 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d7bb69fb-scrzg" event={"ID":"5bb4c5a2-c6c1-4be9-9656-5eecee18ce49","Type":"ContainerDied","Data":"b4fb5565dbdce58fee2f066dacc8a668a17173f0cb30dc2d01d65a91199a9629"} Nov 03 22:54:07.105217 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:07.104796 5084 scope.go:122] "RemoveContainer" containerID="80574e6f20cc41d8e49c6261bc33f056fc1520d8431785ba6e6188c690e790a6" Nov 03 22:54:07.272488 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:07.272445 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57d7bb69fb-scrzg"] Nov 03 22:54:07.323483 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:07.323440 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57d7bb69fb-scrzg"] Nov 03 22:54:08.134127 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.134082 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57d7bb69fb-996cj"] Nov 03 22:54:08.134577 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.134544 5084 cpu_manager.go:414] "containerMap: removing container" podUID="5bb4c5a2-c6c1-4be9-9656-5eecee18ce49" containerName="manager" Nov 03 22:54:08.134577 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.134556 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb4c5a2-c6c1-4be9-9656-5eecee18ce49" containerName="manager" Nov 03 22:54:08.134577 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.134576 5084 cpu_manager.go:414] "containerMap: removing container" podUID="5bb4c5a2-c6c1-4be9-9656-5eecee18ce49" containerName="manager" Nov 03 22:54:08.134703 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.134581 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb4c5a2-c6c1-4be9-9656-5eecee18ce49" containerName="manager" Nov 03 22:54:08.134703 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.134591 5084 cpu_manager.go:414] "containerMap: removing container" podUID="416d4945-1652-4bb1-996d-db1e4d2a8edd" containerName="revoke-keys" Nov 03 22:54:08.134703 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.134597 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416d4945-1652-4bb1-996d-db1e4d2a8edd" containerName="revoke-keys" Nov 03 22:54:08.134703 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.134676 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="416d4945-1652-4bb1-996d-db1e4d2a8edd" containerName="revoke-keys" Nov 03 22:54:08.134703 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.134688 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb4c5a2-c6c1-4be9-9656-5eecee18ce49" containerName="manager" Nov 03 22:54:08.134703 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.134699 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb4c5a2-c6c1-4be9-9656-5eecee18ce49" containerName="manager" Nov 03 22:54:08.139492 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.139463 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d7bb69fb-996cj" Nov 03 22:54:08.193533 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.193499 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 22:54:08.244889 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.244852 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 22:54:08.245191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.244852 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lh6b2\"" Nov 03 22:54:08.245191 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.244881 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 22:54:08.247665 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.247637 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57d7bb69fb-996cj"] Nov 03 22:54:08.275730 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.275694 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-webhook-cert\") pod \"maas-controller-57d7bb69fb-996cj\" (UID: \"8e8fa07f-23ee-40ae-b4da-7467dc9decf4\") " pod="opendatahub/maas-controller-57d7bb69fb-996cj" Nov 03 22:54:08.275730 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.275735 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z76k9\" (UniqueName: \"kubernetes.io/projected/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-kube-api-access-z76k9\") pod \"maas-controller-57d7bb69fb-996cj\" (UID: \"8e8fa07f-23ee-40ae-b4da-7467dc9decf4\") " pod="opendatahub/maas-controller-57d7bb69fb-996cj" Nov 03 22:54:08.275961 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.275884 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-metrics-certs\") pod \"maas-controller-57d7bb69fb-996cj\" (UID: \"8e8fa07f-23ee-40ae-b4da-7467dc9decf4\") " pod="opendatahub/maas-controller-57d7bb69fb-996cj" Nov 03 22:54:08.377242 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.377198 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-webhook-cert\") pod \"maas-controller-57d7bb69fb-996cj\" (UID: \"8e8fa07f-23ee-40ae-b4da-7467dc9decf4\") " pod="opendatahub/maas-controller-57d7bb69fb-996cj" Nov 03 22:54:08.377242 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.377244 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z76k9\" (UniqueName: \"kubernetes.io/projected/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-kube-api-access-z76k9\") pod \"maas-controller-57d7bb69fb-996cj\" (UID: \"8e8fa07f-23ee-40ae-b4da-7467dc9decf4\") " pod="opendatahub/maas-controller-57d7bb69fb-996cj" Nov 03 22:54:08.377488 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.377317 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-metrics-certs\") pod \"maas-controller-57d7bb69fb-996cj\" (UID: \"8e8fa07f-23ee-40ae-b4da-7467dc9decf4\") " pod="opendatahub/maas-controller-57d7bb69fb-996cj" Nov 03 22:54:08.380055 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.380016 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-metrics-certs\") pod \"maas-controller-57d7bb69fb-996cj\" (UID: \"8e8fa07f-23ee-40ae-b4da-7467dc9decf4\") " pod="opendatahub/maas-controller-57d7bb69fb-996cj" Nov 03 22:54:08.380055 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.380042 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-webhook-cert\") pod \"maas-controller-57d7bb69fb-996cj\" (UID: \"8e8fa07f-23ee-40ae-b4da-7467dc9decf4\") " pod="opendatahub/maas-controller-57d7bb69fb-996cj" Nov 03 22:54:08.436219 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.436136 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z76k9\" (UniqueName: \"kubernetes.io/projected/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-kube-api-access-z76k9\") pod \"maas-controller-57d7bb69fb-996cj\" (UID: \"8e8fa07f-23ee-40ae-b4da-7467dc9decf4\") " pod="opendatahub/maas-controller-57d7bb69fb-996cj" Nov 03 22:54:08.453987 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.453939 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d7bb69fb-996cj" Nov 03 22:54:08.629118 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.629076 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb4c5a2-c6c1-4be9-9656-5eecee18ce49" path="/var/lib/kubelet/pods/5bb4c5a2-c6c1-4be9-9656-5eecee18ce49/volumes" Nov 03 22:54:08.825233 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:54:08.825191 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8fa07f_23ee_40ae_b4da_7467dc9decf4.slice/crio-b4c230b5e9f12e89b01241db18d97d3f147d5562cd6b5232b431ba50bad709a5 WatchSource:0}: Error finding container b4c230b5e9f12e89b01241db18d97d3f147d5562cd6b5232b431ba50bad709a5: Status 404 returned error can't find the container with id b4c230b5e9f12e89b01241db18d97d3f147d5562cd6b5232b431ba50bad709a5 Nov 03 22:54:08.826674 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.826649 5084 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:54:08.827553 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:08.827524 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57d7bb69fb-996cj"] Nov 03 22:54:09.123871 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:09.123757 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d7bb69fb-996cj" event={"ID":"8e8fa07f-23ee-40ae-b4da-7467dc9decf4","Type":"ContainerStarted","Data":"b4c230b5e9f12e89b01241db18d97d3f147d5562cd6b5232b431ba50bad709a5"} Nov 03 22:54:10.141502 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:10.141444 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d7bb69fb-996cj" event={"ID":"8e8fa07f-23ee-40ae-b4da-7467dc9decf4","Type":"ContainerStarted","Data":"6987c9dbf7a7c9298961617039f9a01458fd8e3706bc9bdbda10718e896ff550"} Nov 03 22:54:10.142035 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:10.141532 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57d7bb69fb-996cj" Nov 03 22:54:11.328089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:11.327584 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:54:11.328089 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:11.327663 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" podUID="8cebf3c1-c5fb-4feb-beac-7cf2e8a01134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:54:12.025131 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:54:12.025092 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:54:17.062198 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:17.062153 5084 scope.go:122] "RemoveContainer" containerID="4fe9b37e4c339022f8d2c9f323878cd25f6f6ecfcccdff749318d48ebe12d92e" Nov 03 22:54:21.151185 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:21.151150 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57d7bb69fb-996cj" Nov 03 22:54:21.269021 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:21.268957 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-57d7bb69fb-996cj" podStartSLOduration=12.35295302 podStartE2EDuration="13.268943916s" totalImagesPullingTime="915.990896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:10.282864231 +0000 UTC m=+2817.876896206" watchObservedRunningTime="2026-11-03 22:54:21.268943916 +0000 UTC m=+2828.862975894" Nov 03 22:54:41.977516 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:41.977474 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57d7bb69fb-996cj"] Nov 03 22:54:41.978178 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:41.977750 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-57d7bb69fb-996cj" podUID="8e8fa07f-23ee-40ae-b4da-7467dc9decf4" containerName="manager" containerID="cri-o://6987c9dbf7a7c9298961617039f9a01458fd8e3706bc9bdbda10718e896ff550" gracePeriod=30 Nov 03 22:54:42.022984 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:54:42.022945 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:54:42.246354 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.246325 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d7bb69fb-996cj" Nov 03 22:54:42.327764 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.327731 5084 generic.go:356] "Generic (PLEG): container finished" podID="8e8fa07f-23ee-40ae-b4da-7467dc9decf4" containerID="6987c9dbf7a7c9298961617039f9a01458fd8e3706bc9bdbda10718e896ff550" exitCode=0 Nov 03 22:54:42.327946 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.327778 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d7bb69fb-996cj" Nov 03 22:54:42.327946 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.327814 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d7bb69fb-996cj" event={"ID":"8e8fa07f-23ee-40ae-b4da-7467dc9decf4","Type":"ContainerDied","Data":"6987c9dbf7a7c9298961617039f9a01458fd8e3706bc9bdbda10718e896ff550"} Nov 03 22:54:42.327946 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.327869 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d7bb69fb-996cj" event={"ID":"8e8fa07f-23ee-40ae-b4da-7467dc9decf4","Type":"ContainerDied","Data":"b4c230b5e9f12e89b01241db18d97d3f147d5562cd6b5232b431ba50bad709a5"} Nov 03 22:54:42.327946 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.327896 5084 scope.go:122] "RemoveContainer" containerID="6987c9dbf7a7c9298961617039f9a01458fd8e3706bc9bdbda10718e896ff550" Nov 03 22:54:42.337567 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.337541 5084 scope.go:122] "RemoveContainer" containerID="6987c9dbf7a7c9298961617039f9a01458fd8e3706bc9bdbda10718e896ff550" Nov 03 22:54:42.337911 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:54:42.337890 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6987c9dbf7a7c9298961617039f9a01458fd8e3706bc9bdbda10718e896ff550\": container with ID starting with 6987c9dbf7a7c9298961617039f9a01458fd8e3706bc9bdbda10718e896ff550 not found: ID does not exist" containerID="6987c9dbf7a7c9298961617039f9a01458fd8e3706bc9bdbda10718e896ff550" Nov 03 22:54:42.337963 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.337922 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6987c9dbf7a7c9298961617039f9a01458fd8e3706bc9bdbda10718e896ff550"} err="failed to get container status \"6987c9dbf7a7c9298961617039f9a01458fd8e3706bc9bdbda10718e896ff550\": rpc error: code = NotFound desc = could not find container \"6987c9dbf7a7c9298961617039f9a01458fd8e3706bc9bdbda10718e896ff550\": container with ID starting with 6987c9dbf7a7c9298961617039f9a01458fd8e3706bc9bdbda10718e896ff550 not found: ID does not exist" Nov 03 22:54:42.344334 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.344300 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-webhook-cert\") pod \"8e8fa07f-23ee-40ae-b4da-7467dc9decf4\" (UID: \"8e8fa07f-23ee-40ae-b4da-7467dc9decf4\") " Nov 03 22:54:42.344450 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.344354 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-metrics-certs\") pod \"8e8fa07f-23ee-40ae-b4da-7467dc9decf4\" (UID: \"8e8fa07f-23ee-40ae-b4da-7467dc9decf4\") " Nov 03 22:54:42.344450 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.344399 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-kube-api-access-z76k9\" (UniqueName: \"kubernetes.io/projected/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-kube-api-access-z76k9\") pod \"8e8fa07f-23ee-40ae-b4da-7467dc9decf4\" (UID: \"8e8fa07f-23ee-40ae-b4da-7467dc9decf4\") " Nov 03 22:54:42.346613 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.346583 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-webhook-cert" pod "8e8fa07f-23ee-40ae-b4da-7467dc9decf4" (UID: "8e8fa07f-23ee-40ae-b4da-7467dc9decf4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:42.346730 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.346623 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-kube-api-access-z76k9" pod "8e8fa07f-23ee-40ae-b4da-7467dc9decf4" (UID: "8e8fa07f-23ee-40ae-b4da-7467dc9decf4"). InnerVolumeSpecName "kube-api-access-z76k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:42.346730 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.346627 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-metrics-certs" pod "8e8fa07f-23ee-40ae-b4da-7467dc9decf4" (UID: "8e8fa07f-23ee-40ae-b4da-7467dc9decf4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:42.445470 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.445430 5084 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-webhook-cert\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:42.445470 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.445467 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:42.445470 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.445477 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z76k9\" (UniqueName: \"kubernetes.io/projected/8e8fa07f-23ee-40ae-b4da-7467dc9decf4-kube-api-access-z76k9\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 22:54:42.754893 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.754855 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57d7bb69fb-996cj"] Nov 03 22:54:42.758488 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:42.758439 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57d7bb69fb-996cj"] Nov 03 22:54:44.063115 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.063076 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57d7bb69fb-gj9mx"] Nov 03 22:54:44.063623 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.063603 5084 cpu_manager.go:414] "containerMap: removing container" podUID="5bb4c5a2-c6c1-4be9-9656-5eecee18ce49" containerName="manager" Nov 03 22:54:44.063681 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.063625 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb4c5a2-c6c1-4be9-9656-5eecee18ce49" containerName="manager" Nov 03 22:54:44.063681 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.063650 5084 cpu_manager.go:414] "containerMap: removing container" podUID="8e8fa07f-23ee-40ae-b4da-7467dc9decf4" containerName="manager" Nov 03 22:54:44.063681 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.063655 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8fa07f-23ee-40ae-b4da-7467dc9decf4" containerName="manager" Nov 03 22:54:44.063775 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.063735 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8fa07f-23ee-40ae-b4da-7467dc9decf4" containerName="manager" Nov 03 22:54:44.063775 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.063745 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb4c5a2-c6c1-4be9-9656-5eecee18ce49" containerName="manager" Nov 03 22:54:44.068292 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.068269 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d7bb69fb-gj9mx" Nov 03 22:54:44.163209 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.163168 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5z8f\" (UniqueName: \"kubernetes.io/projected/a1870f1b-ca5c-4885-aecd-710452f04516-kube-api-access-k5z8f\") pod \"maas-controller-57d7bb69fb-gj9mx\" (UID: \"a1870f1b-ca5c-4885-aecd-710452f04516\") " pod="opendatahub/maas-controller-57d7bb69fb-gj9mx" Nov 03 22:54:44.163427 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.163250 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1870f1b-ca5c-4885-aecd-710452f04516-webhook-cert\") pod \"maas-controller-57d7bb69fb-gj9mx\" (UID: \"a1870f1b-ca5c-4885-aecd-710452f04516\") " pod="opendatahub/maas-controller-57d7bb69fb-gj9mx" Nov 03 22:54:44.163427 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.163365 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1870f1b-ca5c-4885-aecd-710452f04516-metrics-certs\") pod \"maas-controller-57d7bb69fb-gj9mx\" (UID: \"a1870f1b-ca5c-4885-aecd-710452f04516\") " pod="opendatahub/maas-controller-57d7bb69fb-gj9mx" Nov 03 22:54:44.173325 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.173275 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 22:54:44.173325 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.173285 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 22:54:44.173616 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.173367 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 22:54:44.173616 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.173415 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lh6b2\"" Nov 03 22:54:44.176156 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.176128 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57d7bb69fb-gj9mx"] Nov 03 22:54:44.264730 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.264690 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5z8f\" (UniqueName: \"kubernetes.io/projected/a1870f1b-ca5c-4885-aecd-710452f04516-kube-api-access-k5z8f\") pod \"maas-controller-57d7bb69fb-gj9mx\" (UID: \"a1870f1b-ca5c-4885-aecd-710452f04516\") " pod="opendatahub/maas-controller-57d7bb69fb-gj9mx" Nov 03 22:54:44.264910 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.264851 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1870f1b-ca5c-4885-aecd-710452f04516-webhook-cert\") pod \"maas-controller-57d7bb69fb-gj9mx\" (UID: \"a1870f1b-ca5c-4885-aecd-710452f04516\") " pod="opendatahub/maas-controller-57d7bb69fb-gj9mx" Nov 03 22:54:44.264963 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.264934 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1870f1b-ca5c-4885-aecd-710452f04516-metrics-certs\") pod \"maas-controller-57d7bb69fb-gj9mx\" (UID: \"a1870f1b-ca5c-4885-aecd-710452f04516\") " pod="opendatahub/maas-controller-57d7bb69fb-gj9mx" Nov 03 22:54:44.267499 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.267474 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1870f1b-ca5c-4885-aecd-710452f04516-metrics-certs\") pod \"maas-controller-57d7bb69fb-gj9mx\" (UID: \"a1870f1b-ca5c-4885-aecd-710452f04516\") " pod="opendatahub/maas-controller-57d7bb69fb-gj9mx" Nov 03 22:54:44.267616 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.267520 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1870f1b-ca5c-4885-aecd-710452f04516-webhook-cert\") pod \"maas-controller-57d7bb69fb-gj9mx\" (UID: \"a1870f1b-ca5c-4885-aecd-710452f04516\") " pod="opendatahub/maas-controller-57d7bb69fb-gj9mx" Nov 03 22:54:44.322627 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.322536 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5z8f\" (UniqueName: \"kubernetes.io/projected/a1870f1b-ca5c-4885-aecd-710452f04516-kube-api-access-k5z8f\") pod \"maas-controller-57d7bb69fb-gj9mx\" (UID: \"a1870f1b-ca5c-4885-aecd-710452f04516\") " pod="opendatahub/maas-controller-57d7bb69fb-gj9mx" Nov 03 22:54:44.383255 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.383217 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d7bb69fb-gj9mx" Nov 03 22:54:44.627394 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.627309 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e8fa07f-23ee-40ae-b4da-7467dc9decf4" path="/var/lib/kubelet/pods/8e8fa07f-23ee-40ae-b4da-7467dc9decf4/volumes" Nov 03 22:54:44.751957 ip-10-0-141-84 kubenswrapper[5084]: W1103 22:54:44.751920 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1870f1b_ca5c_4885_aecd_710452f04516.slice/crio-e15289d967a1956e0cfda3f5841712105970b727d90b3e90715a27373985d5c2 WatchSource:0}: Error finding container e15289d967a1956e0cfda3f5841712105970b727d90b3e90715a27373985d5c2: Status 404 returned error can't find the container with id e15289d967a1956e0cfda3f5841712105970b727d90b3e90715a27373985d5c2 Nov 03 22:54:44.754557 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:44.754460 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57d7bb69fb-gj9mx"] Nov 03 22:54:45.347396 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:45.347358 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d7bb69fb-gj9mx" event={"ID":"a1870f1b-ca5c-4885-aecd-710452f04516","Type":"ContainerStarted","Data":"e15289d967a1956e0cfda3f5841712105970b727d90b3e90715a27373985d5c2"} Nov 03 22:54:46.355119 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:46.355056 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d7bb69fb-gj9mx" event={"ID":"a1870f1b-ca5c-4885-aecd-710452f04516","Type":"ContainerStarted","Data":"05d3f52aa3aebc199ff6f724a5362a5cd91199f7500cf6ee25d2447db7417d0d"} Nov 03 22:54:46.355524 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:46.355284 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57d7bb69fb-gj9mx" Nov 03 22:54:46.472104 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:46.472033 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-57d7bb69fb-gj9mx" podStartSLOduration=1.560981912 podStartE2EDuration="2.472016218s" totalImagesPullingTime="911.034306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:46.471264686 +0000 UTC m=+2854.065296659" watchObservedRunningTime="2026-11-03 22:54:46.472016218 +0000 UTC m=+2854.066048193" Nov 03 22:54:57.364948 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:54:57.364909 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57d7bb69fb-gj9mx" Nov 03 22:55:12.024871 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:55:12.024829 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:55:42.021835 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:55:42.021792 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:55:57.207153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:55:57.206194 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-l96p9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:55:57.207153 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:55:57.206273 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" podUID="d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:56:12.024144 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:56:12.024106 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:56:41.328882 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:56:41.328838 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:56:41.329507 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:56:41.329473 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" podUID="8cebf3c1-c5fb-4feb-beac-7cf2e8a01134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:56:42.024104 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:56:42.024034 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:57:12.032090 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:57:12.031993 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:57:13.259091 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:57:13.255170 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:57:13.314108 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:57:13.314005 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 22:57:42.022212 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:57:42.022171 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:57:50.052602 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:57:50.052561 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx"] Nov 03 22:57:50.103543 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:57:50.103501 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k"] Nov 03 22:57:50.106341 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:57:50.106308 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8d76e57-g22tx"] Nov 03 22:57:50.154826 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:57:50.154783 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4236a363-fkm6k"] Nov 03 22:57:50.628301 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:57:50.628265 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed3f9cad-3ed7-440f-b9e3-b3371d649bb8" path="/var/lib/kubelet/pods/ed3f9cad-3ed7-440f-b9e3-b3371d649bb8/volumes" Nov 03 22:57:50.628726 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:57:50.628712 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef5ac70-c9c9-46ad-aadb-7f5a54264240" path="/var/lib/kubelet/pods/fef5ac70-c9c9-46ad-aadb-7f5a54264240/volumes" Nov 03 22:58:12.025567 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:58:12.025524 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:58:40.047538 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:58:40.047480 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26"] Nov 03 22:58:40.098499 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:58:40.098460 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac0d55ff-94l26"] Nov 03 22:58:40.631846 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:58:40.631807 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="416d4945-1652-4bb1-996d-db1e4d2a8edd" path="/var/lib/kubelet/pods/416d4945-1652-4bb1-996d-db1e4d2a8edd/volumes" Nov 03 22:58:42.022433 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:58:42.022396 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:59:12.024557 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:59:12.024517 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 22:59:17.150462 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:59:17.150430 5084 scope.go:122] "RemoveContainer" containerID="5d86de75a304eb49af29e388a84aa327bc42627394297f07aedcdbd5ddb67533" Nov 03 22:59:17.230963 ip-10-0-141-84 kubenswrapper[5084]: I1103 22:59:17.228654 5084 scope.go:122] "RemoveContainer" containerID="01c84796e70bd9ec6e6a4d39f1a56c3634762419b4084c4661ee2ce538c0b499" Nov 03 22:59:42.022950 ip-10-0-141-84 kubenswrapper[5084]: E1103 22:59:42.022905 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:00:11.098606 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:11.098557 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs"] Nov 03 23:00:11.102388 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:11.102360 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs" Nov 03 23:00:11.155820 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:11.155780 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs"] Nov 03 23:00:11.203728 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:11.203688 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:00:11.203728 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:11.203700 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:00:11.206554 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:11.206526 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:00:11.206554 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:11.206556 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rhm9d\"" Nov 03 23:00:11.210236 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:11.210206 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff937d3a-685d-4e82-a425-50b82ba33369-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-vqtgs\" (UID: \"ff937d3a-685d-4e82-a425-50b82ba33369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs" Nov 03 23:00:11.210339 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:11.210259 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff937d3a-685d-4e82-a425-50b82ba33369-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-vqtgs\" (UID: \"ff937d3a-685d-4e82-a425-50b82ba33369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs" Nov 03 23:00:11.210380 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:11.210364 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkcck\" (UniqueName: \"kubernetes.io/projected/ff937d3a-685d-4e82-a425-50b82ba33369-kube-api-access-tkcck\") pod \"maas-api-revoke-sub-e2fc9c5b-vqtgs\" (UID: \"ff937d3a-685d-4e82-a425-50b82ba33369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs" Nov 03 23:00:11.311719 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:11.311672 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkcck\" (UniqueName: \"kubernetes.io/projected/ff937d3a-685d-4e82-a425-50b82ba33369-kube-api-access-tkcck\") pod \"maas-api-revoke-sub-e2fc9c5b-vqtgs\" (UID: \"ff937d3a-685d-4e82-a425-50b82ba33369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs" Nov 03 23:00:11.311944 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:11.311786 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff937d3a-685d-4e82-a425-50b82ba33369-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-vqtgs\" (UID: \"ff937d3a-685d-4e82-a425-50b82ba33369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs" Nov 03 23:00:11.311944 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:11.311826 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff937d3a-685d-4e82-a425-50b82ba33369-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-vqtgs\" (UID: \"ff937d3a-685d-4e82-a425-50b82ba33369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs" Nov 03 23:00:11.312625 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:11.312596 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff937d3a-685d-4e82-a425-50b82ba33369-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-vqtgs\" (UID: \"ff937d3a-685d-4e82-a425-50b82ba33369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs" Nov 03 23:00:11.314380 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:11.314351 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff937d3a-685d-4e82-a425-50b82ba33369-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-vqtgs\" (UID: \"ff937d3a-685d-4e82-a425-50b82ba33369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs" Nov 03 23:00:11.369919 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:11.369827 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkcck\" (UniqueName: \"kubernetes.io/projected/ff937d3a-685d-4e82-a425-50b82ba33369-kube-api-access-tkcck\") pod \"maas-api-revoke-sub-e2fc9c5b-vqtgs\" (UID: \"ff937d3a-685d-4e82-a425-50b82ba33369\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs" Nov 03 23:00:11.416621 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:11.416575 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs" Nov 03 23:00:11.797525 ip-10-0-141-84 kubenswrapper[5084]: W1103 23:00:11.797490 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff937d3a_685d_4e82_a425_50b82ba33369.slice/crio-34f60f71c224090ee83c43ee865f081365f503dd736eaba3664853c8fac11a89 WatchSource:0}: Error finding container 34f60f71c224090ee83c43ee865f081365f503dd736eaba3664853c8fac11a89: Status 404 returned error can't find the container with id 34f60f71c224090ee83c43ee865f081365f503dd736eaba3664853c8fac11a89 Nov 03 23:00:11.799899 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:11.799870 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs"] Nov 03 23:00:12.026516 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:00:12.026479 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:00:12.486804 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:12.486707 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs" event={"ID":"ff937d3a-685d-4e82-a425-50b82ba33369","Type":"ContainerStarted","Data":"e047655352623685837b25076a154336c5cb6954608893f595644d77461685cd"} Nov 03 23:00:12.486804 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:12.486758 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs" event={"ID":"ff937d3a-685d-4e82-a425-50b82ba33369","Type":"ContainerStarted","Data":"34f60f71c224090ee83c43ee865f081365f503dd736eaba3664853c8fac11a89"} Nov 03 23:00:12.677536 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:12.677468 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs" podStartSLOduration=1.677450439 podStartE2EDuration="1.677450439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:12.633836439 +0000 UTC m=+3180.227868415" watchObservedRunningTime="2026-11-03 23:00:12.677450439 +0000 UTC m=+3180.271482472" Nov 03 23:00:13.494574 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:13.494536 5084 generic.go:356] "Generic (PLEG): container finished" podID="ff937d3a-685d-4e82-a425-50b82ba33369" containerID="e047655352623685837b25076a154336c5cb6954608893f595644d77461685cd" exitCode=0 Nov 03 23:00:13.494968 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:13.494629 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs" event={"ID":"ff937d3a-685d-4e82-a425-50b82ba33369","Type":"ContainerDied","Data":"e047655352623685837b25076a154336c5cb6954608893f595644d77461685cd"} Nov 03 23:00:14.646309 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:14.646277 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs" Nov 03 23:00:14.749802 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:14.749761 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff937d3a-685d-4e82-a425-50b82ba33369-kube-api-access-tkcck\" (UniqueName: \"kubernetes.io/projected/ff937d3a-685d-4e82-a425-50b82ba33369-kube-api-access-tkcck\") pod \"ff937d3a-685d-4e82-a425-50b82ba33369\" (UID: \"ff937d3a-685d-4e82-a425-50b82ba33369\") " Nov 03 23:00:14.749995 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:14.749815 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff937d3a-685d-4e82-a425-50b82ba33369-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff937d3a-685d-4e82-a425-50b82ba33369-maas-api-openshift-ca\") pod \"ff937d3a-685d-4e82-a425-50b82ba33369\" (UID: \"ff937d3a-685d-4e82-a425-50b82ba33369\") " Nov 03 23:00:14.749995 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:14.749838 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff937d3a-685d-4e82-a425-50b82ba33369-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff937d3a-685d-4e82-a425-50b82ba33369-maas-api-xks-ca\") pod \"ff937d3a-685d-4e82-a425-50b82ba33369\" (UID: \"ff937d3a-685d-4e82-a425-50b82ba33369\") " Nov 03 23:00:14.750425 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:14.750391 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff937d3a-685d-4e82-a425-50b82ba33369-maas-api-openshift-ca" pod "ff937d3a-685d-4e82-a425-50b82ba33369" (UID: "ff937d3a-685d-4e82-a425-50b82ba33369"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:14.752736 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:14.752661 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff937d3a-685d-4e82-a425-50b82ba33369-maas-api-xks-ca" pod "ff937d3a-685d-4e82-a425-50b82ba33369" (UID: "ff937d3a-685d-4e82-a425-50b82ba33369"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:14.752736 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:14.752665 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff937d3a-685d-4e82-a425-50b82ba33369-kube-api-access-tkcck" pod "ff937d3a-685d-4e82-a425-50b82ba33369" (UID: "ff937d3a-685d-4e82-a425-50b82ba33369"). InnerVolumeSpecName "kube-api-access-tkcck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:14.850981 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:14.850935 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkcck\" (UniqueName: \"kubernetes.io/projected/ff937d3a-685d-4e82-a425-50b82ba33369-kube-api-access-tkcck\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:14.850981 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:14.850977 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff937d3a-685d-4e82-a425-50b82ba33369-maas-api-openshift-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:14.851223 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:14.850993 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff937d3a-685d-4e82-a425-50b82ba33369-maas-api-xks-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:00:15.508819 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:15.508785 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs" Nov 03 23:00:15.509021 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:15.508814 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs" event={"ID":"ff937d3a-685d-4e82-a425-50b82ba33369","Type":"ContainerDied","Data":"34f60f71c224090ee83c43ee865f081365f503dd736eaba3664853c8fac11a89"} Nov 03 23:00:15.509021 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:15.508875 5084 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34f60f71c224090ee83c43ee865f081365f503dd736eaba3664853c8fac11a89" Nov 03 23:00:17.344284 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:17.344248 5084 scope.go:122] "RemoveContainer" containerID="3a14533ecbc70bf0cd7f83075c5cc8768e6ddf0754bbbf4206cfa6e921512fd0" Nov 03 23:00:26.313436 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:26.313324 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:00:26.314244 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:00:26.314190 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" podUID="8cebf3c1-c5fb-4feb-beac-7cf2e8a01134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:00:42.022533 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:00:42.022488 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:01:06.121114 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.121058 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w"] Nov 03 23:01:06.121637 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.121526 5084 cpu_manager.go:414] "containerMap: removing container" podUID="ff937d3a-685d-4e82-a425-50b82ba33369" containerName="revoke-keys" Nov 03 23:01:06.121637 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.121539 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff937d3a-685d-4e82-a425-50b82ba33369" containerName="revoke-keys" Nov 03 23:01:06.121637 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.121624 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff937d3a-685d-4e82-a425-50b82ba33369" containerName="revoke-keys" Nov 03 23:01:06.124959 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.124936 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:01:06.178233 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.178198 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w"] Nov 03 23:01:06.223472 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.223428 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bdcz\" (UniqueName: \"kubernetes.io/projected/a25953d9-0c0e-4af1-85aa-c32add32d122-kube-api-access-5bdcz\") pod \"maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w\" (UID: \"a25953d9-0c0e-4af1-85aa-c32add32d122\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:01:06.223686 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.223484 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a25953d9-0c0e-4af1-85aa-c32add32d122-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w\" (UID: \"a25953d9-0c0e-4af1-85aa-c32add32d122\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:01:06.223686 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.223612 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25953d9-0c0e-4af1-85aa-c32add32d122-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w\" (UID: \"a25953d9-0c0e-4af1-85aa-c32add32d122\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:01:06.223774 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.223683 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a25953d9-0c0e-4af1-85aa-c32add32d122-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w\" (UID: \"a25953d9-0c0e-4af1-85aa-c32add32d122\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:01:06.226620 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.226590 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:01:06.226620 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.226608 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-fe3af2-serving-cert\"" Nov 03 23:01:06.226838 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.226724 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:01:06.229326 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.229294 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vjv49\"" Nov 03 23:01:06.229326 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.229294 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 23:01:06.324561 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.324517 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bdcz\" (UniqueName: \"kubernetes.io/projected/a25953d9-0c0e-4af1-85aa-c32add32d122-kube-api-access-5bdcz\") pod \"maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w\" (UID: \"a25953d9-0c0e-4af1-85aa-c32add32d122\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:01:06.324771 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.324571 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a25953d9-0c0e-4af1-85aa-c32add32d122-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w\" (UID: \"a25953d9-0c0e-4af1-85aa-c32add32d122\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:01:06.324771 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:01:06.324667 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-serving-cert: secret "maas-api-e2e-worker-single-main-fe3af2-serving-cert" not found Nov 03 23:01:06.324771 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.324694 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25953d9-0c0e-4af1-85aa-c32add32d122-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w\" (UID: \"a25953d9-0c0e-4af1-85aa-c32add32d122\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:01:06.324771 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:01:06.324734 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a25953d9-0c0e-4af1-85aa-c32add32d122-maas-api-tls podName:a25953d9-0c0e-4af1-85aa-c32add32d122 nodeName:}" failed. No retries permitted until 2026-11-03 23:01:06.824715635 +0000 UTC m=+3234.418747617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a25953d9-0c0e-4af1-85aa-c32add32d122-maas-api-tls") pod "maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" (UID: "a25953d9-0c0e-4af1-85aa-c32add32d122") : secret "maas-api-e2e-worker-single-main-fe3af2-serving-cert" not found Nov 03 23:01:06.324771 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.324767 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a25953d9-0c0e-4af1-85aa-c32add32d122-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w\" (UID: \"a25953d9-0c0e-4af1-85aa-c32add32d122\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:01:06.325635 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.325609 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a25953d9-0c0e-4af1-85aa-c32add32d122-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w\" (UID: \"a25953d9-0c0e-4af1-85aa-c32add32d122\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:01:06.327437 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.327412 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25953d9-0c0e-4af1-85aa-c32add32d122-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w\" (UID: \"a25953d9-0c0e-4af1-85aa-c32add32d122\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:01:06.387168 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.386978 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bdcz\" (UniqueName: \"kubernetes.io/projected/a25953d9-0c0e-4af1-85aa-c32add32d122-kube-api-access-5bdcz\") pod \"maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w\" (UID: \"a25953d9-0c0e-4af1-85aa-c32add32d122\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:01:06.830151 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.830056 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a25953d9-0c0e-4af1-85aa-c32add32d122-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w\" (UID: \"a25953d9-0c0e-4af1-85aa-c32add32d122\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:01:06.832798 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:06.832770 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a25953d9-0c0e-4af1-85aa-c32add32d122-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w\" (UID: \"a25953d9-0c0e-4af1-85aa-c32add32d122\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:01:07.039007 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:07.038965 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:01:07.412689 ip-10-0-141-84 kubenswrapper[5084]: W1103 23:01:07.412643 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25953d9_0c0e_4af1_85aa_c32add32d122.slice/crio-c9e5d4e2eb2de19e8e100780e25cf42f351b50090570e10773f8c530ab4af96e WatchSource:0}: Error finding container c9e5d4e2eb2de19e8e100780e25cf42f351b50090570e10773f8c530ab4af96e: Status 404 returned error can't find the container with id c9e5d4e2eb2de19e8e100780e25cf42f351b50090570e10773f8c530ab4af96e Nov 03 23:01:07.414388 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:07.414359 5084 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:01:07.415132 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:07.415106 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w"] Nov 03 23:01:07.871464 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:07.871405 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" event={"ID":"a25953d9-0c0e-4af1-85aa-c32add32d122","Type":"ContainerStarted","Data":"c9e5d4e2eb2de19e8e100780e25cf42f351b50090570e10773f8c530ab4af96e"} Nov 03 23:01:08.880165 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:08.880123 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" event={"ID":"a25953d9-0c0e-4af1-85aa-c32add32d122","Type":"ContainerStarted","Data":"ebb16aea08310a2eb7c0dd3f4cf21834289969dd6c0741595cf6f073b1d3ebb7"} Nov 03 23:01:08.880638 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:08.880200 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:01:09.050398 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:09.050328 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" podStartSLOduration=2.189195729 podStartE2EDuration="3.050311972s" totalImagesPullingTime="861.116243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:08.997221481 +0000 UTC m=+3236.591253457" watchObservedRunningTime="2026-11-03 23:01:09.050311972 +0000 UTC m=+3236.644343939" Nov 03 23:01:12.024509 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:01:12.024468 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:01:14.892326 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:01:14.892284 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:01:42.022329 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:01:42.022289 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:02:12.024118 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:02:12.024055 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:02:13.376617 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:13.376591 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 23:02:13.378335 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:13.378305 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 23:02:35.716033 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:35.715988 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w"] Nov 03 23:02:35.716504 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:35.716214 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" podUID="a25953d9-0c0e-4af1-85aa-c32add32d122" containerName="maas-api" containerID="cri-o://ebb16aea08310a2eb7c0dd3f4cf21834289969dd6c0741595cf6f073b1d3ebb7" gracePeriod=30 Nov 03 23:02:36.985496 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:36.985468 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:02:37.076720 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.076606 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a25953d9-0c0e-4af1-85aa-c32add32d122-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a25953d9-0c0e-4af1-85aa-c32add32d122-maas-api-tls\") pod \"a25953d9-0c0e-4af1-85aa-c32add32d122\" (UID: \"a25953d9-0c0e-4af1-85aa-c32add32d122\") " Nov 03 23:02:37.076720 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.076669 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a25953d9-0c0e-4af1-85aa-c32add32d122-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25953d9-0c0e-4af1-85aa-c32add32d122-metrics-certs\") pod \"a25953d9-0c0e-4af1-85aa-c32add32d122\" (UID: \"a25953d9-0c0e-4af1-85aa-c32add32d122\") " Nov 03 23:02:37.076720 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.076699 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a25953d9-0c0e-4af1-85aa-c32add32d122-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a25953d9-0c0e-4af1-85aa-c32add32d122-openshift-service-ca\") pod \"a25953d9-0c0e-4af1-85aa-c32add32d122\" (UID: \"a25953d9-0c0e-4af1-85aa-c32add32d122\") " Nov 03 23:02:37.077029 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.076774 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25953d9-0c0e-4af1-85aa-c32add32d122-kube-api-access-5bdcz\" (UniqueName: \"kubernetes.io/projected/a25953d9-0c0e-4af1-85aa-c32add32d122-kube-api-access-5bdcz\") pod \"a25953d9-0c0e-4af1-85aa-c32add32d122\" (UID: \"a25953d9-0c0e-4af1-85aa-c32add32d122\") " Nov 03 23:02:37.077249 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.077215 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25953d9-0c0e-4af1-85aa-c32add32d122-openshift-service-ca" pod "a25953d9-0c0e-4af1-85aa-c32add32d122" (UID: "a25953d9-0c0e-4af1-85aa-c32add32d122"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:37.079189 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.079151 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a25953d9-0c0e-4af1-85aa-c32add32d122-maas-api-tls" pod "a25953d9-0c0e-4af1-85aa-c32add32d122" (UID: "a25953d9-0c0e-4af1-85aa-c32add32d122"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:37.079311 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.079199 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a25953d9-0c0e-4af1-85aa-c32add32d122-metrics-certs" pod "a25953d9-0c0e-4af1-85aa-c32add32d122" (UID: "a25953d9-0c0e-4af1-85aa-c32add32d122"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:37.079311 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.079219 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25953d9-0c0e-4af1-85aa-c32add32d122-kube-api-access-5bdcz" pod "a25953d9-0c0e-4af1-85aa-c32add32d122" (UID: "a25953d9-0c0e-4af1-85aa-c32add32d122"). InnerVolumeSpecName "kube-api-access-5bdcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:37.178412 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.178367 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a25953d9-0c0e-4af1-85aa-c32add32d122-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:37.178412 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.178403 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a25953d9-0c0e-4af1-85aa-c32add32d122-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:37.178412 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.178415 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a25953d9-0c0e-4af1-85aa-c32add32d122-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:37.178668 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.178424 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bdcz\" (UniqueName: \"kubernetes.io/projected/a25953d9-0c0e-4af1-85aa-c32add32d122-kube-api-access-5bdcz\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:02:37.421777 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.421663 5084 generic.go:356] "Generic (PLEG): container finished" podID="a25953d9-0c0e-4af1-85aa-c32add32d122" containerID="ebb16aea08310a2eb7c0dd3f4cf21834289969dd6c0741595cf6f073b1d3ebb7" exitCode=0 Nov 03 23:02:37.421777 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.421715 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" Nov 03 23:02:37.421777 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.421748 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" event={"ID":"a25953d9-0c0e-4af1-85aa-c32add32d122","Type":"ContainerDied","Data":"ebb16aea08310a2eb7c0dd3f4cf21834289969dd6c0741595cf6f073b1d3ebb7"} Nov 03 23:02:37.422105 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.421815 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w" event={"ID":"a25953d9-0c0e-4af1-85aa-c32add32d122","Type":"ContainerDied","Data":"c9e5d4e2eb2de19e8e100780e25cf42f351b50090570e10773f8c530ab4af96e"} Nov 03 23:02:37.422105 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.421842 5084 scope.go:122] "RemoveContainer" containerID="ebb16aea08310a2eb7c0dd3f4cf21834289969dd6c0741595cf6f073b1d3ebb7" Nov 03 23:02:37.435346 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.435323 5084 scope.go:122] "RemoveContainer" containerID="ebb16aea08310a2eb7c0dd3f4cf21834289969dd6c0741595cf6f073b1d3ebb7" Nov 03 23:02:37.435736 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:02:37.435709 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb16aea08310a2eb7c0dd3f4cf21834289969dd6c0741595cf6f073b1d3ebb7\": container with ID starting with ebb16aea08310a2eb7c0dd3f4cf21834289969dd6c0741595cf6f073b1d3ebb7 not found: ID does not exist" containerID="ebb16aea08310a2eb7c0dd3f4cf21834289969dd6c0741595cf6f073b1d3ebb7" Nov 03 23:02:37.435820 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.435749 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb16aea08310a2eb7c0dd3f4cf21834289969dd6c0741595cf6f073b1d3ebb7"} err="failed to get container status \"ebb16aea08310a2eb7c0dd3f4cf21834289969dd6c0741595cf6f073b1d3ebb7\": rpc error: code = NotFound desc = could not find container \"ebb16aea08310a2eb7c0dd3f4cf21834289969dd6c0741595cf6f073b1d3ebb7\": container with ID starting with ebb16aea08310a2eb7c0dd3f4cf21834289969dd6c0741595cf6f073b1d3ebb7 not found: ID does not exist" Nov 03 23:02:37.589170 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.589132 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w"] Nov 03 23:02:37.641664 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:37.641622 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe3af2-f76754c55-hsw6w"] Nov 03 23:02:38.628337 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:02:38.628297 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a25953d9-0c0e-4af1-85aa-c32add32d122" path="/var/lib/kubelet/pods/a25953d9-0c0e-4af1-85aa-c32add32d122/volumes" Nov 03 23:02:42.023525 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:02:42.023486 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:03:08.480116 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:03:08.480072 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc"] Nov 03 23:03:08.480774 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:03:08.480346 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc" podUID="89dbe468-1bfc-435e-8c1d-62cf0ccf29ef" containerName="manager" containerID="cri-o://9e7e02666b7dc013420100fe1e13ad297b9703ce7f8509e83195987077edc571" gracePeriod=10 Nov 03 23:03:08.638464 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:03:08.638428 5084 generic.go:356] "Generic (PLEG): container finished" podID="89dbe468-1bfc-435e-8c1d-62cf0ccf29ef" containerID="9e7e02666b7dc013420100fe1e13ad297b9703ce7f8509e83195987077edc571" exitCode=0 Nov 03 23:03:08.638663 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:03:08.638485 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc" event={"ID":"89dbe468-1bfc-435e-8c1d-62cf0ccf29ef","Type":"ContainerDied","Data":"9e7e02666b7dc013420100fe1e13ad297b9703ce7f8509e83195987077edc571"} Nov 03 23:03:08.748316 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:03:08.748233 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc" Nov 03 23:03:08.905770 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:03:08.905716 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89dbe468-1bfc-435e-8c1d-62cf0ccf29ef-kube-api-access-w5zr2\" (UniqueName: \"kubernetes.io/projected/89dbe468-1bfc-435e-8c1d-62cf0ccf29ef-kube-api-access-w5zr2\") pod \"89dbe468-1bfc-435e-8c1d-62cf0ccf29ef\" (UID: \"89dbe468-1bfc-435e-8c1d-62cf0ccf29ef\") " Nov 03 23:03:08.905974 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:03:08.905786 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89dbe468-1bfc-435e-8c1d-62cf0ccf29ef-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89dbe468-1bfc-435e-8c1d-62cf0ccf29ef-extensions-socket-volume\") pod \"89dbe468-1bfc-435e-8c1d-62cf0ccf29ef\" (UID: \"89dbe468-1bfc-435e-8c1d-62cf0ccf29ef\") " Nov 03 23:03:08.906302 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:03:08.906257 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89dbe468-1bfc-435e-8c1d-62cf0ccf29ef-extensions-socket-volume" pod "89dbe468-1bfc-435e-8c1d-62cf0ccf29ef" (UID: "89dbe468-1bfc-435e-8c1d-62cf0ccf29ef"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:03:08.908384 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:03:08.908340 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89dbe468-1bfc-435e-8c1d-62cf0ccf29ef-kube-api-access-w5zr2" pod "89dbe468-1bfc-435e-8c1d-62cf0ccf29ef" (UID: "89dbe468-1bfc-435e-8c1d-62cf0ccf29ef"). InnerVolumeSpecName "kube-api-access-w5zr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:09.006750 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:03:09.006647 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5zr2\" (UniqueName: \"kubernetes.io/projected/89dbe468-1bfc-435e-8c1d-62cf0ccf29ef-kube-api-access-w5zr2\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:09.006750 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:03:09.006688 5084 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89dbe468-1bfc-435e-8c1d-62cf0ccf29ef-extensions-socket-volume\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:03:09.645688 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:03:09.645657 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc" Nov 03 23:03:09.646199 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:03:09.645693 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc" event={"ID":"89dbe468-1bfc-435e-8c1d-62cf0ccf29ef","Type":"ContainerDied","Data":"8f8fddbdc9967f9f39d9136300853ac36043c4d181bf44a152e93e53567ee098"} Nov 03 23:03:09.646199 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:03:09.645760 5084 scope.go:122] "RemoveContainer" containerID="9e7e02666b7dc013420100fe1e13ad297b9703ce7f8509e83195987077edc571" Nov 03 23:03:09.864941 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:03:09.864903 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc"] Nov 03 23:03:09.919125 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:03:09.919039 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-fdgfc"] Nov 03 23:03:10.627199 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:03:10.627161 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89dbe468-1bfc-435e-8c1d-62cf0ccf29ef" path="/var/lib/kubelet/pods/89dbe468-1bfc-435e-8c1d-62cf0ccf29ef/volumes" Nov 03 23:03:12.024619 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:03:12.024571 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:03:42.021498 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:03:42.021457 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:04:12.024552 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:04:12.024504 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:04:12.133196 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:12.133152 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v5q4x"] Nov 03 23:04:12.133603 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:12.133589 5084 cpu_manager.go:414] "containerMap: removing container" podUID="a25953d9-0c0e-4af1-85aa-c32add32d122" containerName="maas-api" Nov 03 23:04:12.133649 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:12.133603 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a25953d9-0c0e-4af1-85aa-c32add32d122" containerName="maas-api" Nov 03 23:04:12.133649 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:12.133636 5084 cpu_manager.go:414] "containerMap: removing container" podUID="89dbe468-1bfc-435e-8c1d-62cf0ccf29ef" containerName="manager" Nov 03 23:04:12.133649 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:12.133643 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89dbe468-1bfc-435e-8c1d-62cf0ccf29ef" containerName="manager" Nov 03 23:04:12.133758 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:12.133709 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="a25953d9-0c0e-4af1-85aa-c32add32d122" containerName="maas-api" Nov 03 23:04:12.133758 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:12.133724 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="89dbe468-1bfc-435e-8c1d-62cf0ccf29ef" containerName="manager" Nov 03 23:04:12.136998 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:12.136974 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v5q4x" Nov 03 23:04:12.178464 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:12.178429 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v5q4x"] Nov 03 23:04:12.197077 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:12.197035 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/243c55d2-1d9c-4e22-be54-9360d9f506f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-v5q4x\" (UID: \"243c55d2-1d9c-4e22-be54-9360d9f506f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v5q4x" Nov 03 23:04:12.197305 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:12.197212 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7fq4\" (UniqueName: \"kubernetes.io/projected/243c55d2-1d9c-4e22-be54-9360d9f506f3-kube-api-access-l7fq4\") pod \"kuadrant-operator-controller-manager-678d87766f-v5q4x\" (UID: \"243c55d2-1d9c-4e22-be54-9360d9f506f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v5q4x" Nov 03 23:04:12.226927 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:12.226897 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l662p\"" Nov 03 23:04:12.297788 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:12.297683 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7fq4\" (UniqueName: \"kubernetes.io/projected/243c55d2-1d9c-4e22-be54-9360d9f506f3-kube-api-access-l7fq4\") pod \"kuadrant-operator-controller-manager-678d87766f-v5q4x\" (UID: \"243c55d2-1d9c-4e22-be54-9360d9f506f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v5q4x" Nov 03 23:04:12.297788 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:12.297766 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/243c55d2-1d9c-4e22-be54-9360d9f506f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-v5q4x\" (UID: \"243c55d2-1d9c-4e22-be54-9360d9f506f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v5q4x" Nov 03 23:04:12.298171 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:12.298153 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/243c55d2-1d9c-4e22-be54-9360d9f506f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-v5q4x\" (UID: \"243c55d2-1d9c-4e22-be54-9360d9f506f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v5q4x" Nov 03 23:04:12.356020 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:12.355987 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7fq4\" (UniqueName: \"kubernetes.io/projected/243c55d2-1d9c-4e22-be54-9360d9f506f3-kube-api-access-l7fq4\") pod \"kuadrant-operator-controller-manager-678d87766f-v5q4x\" (UID: \"243c55d2-1d9c-4e22-be54-9360d9f506f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v5q4x" Nov 03 23:04:12.452858 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:12.452814 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v5q4x" Nov 03 23:04:12.921457 ip-10-0-141-84 kubenswrapper[5084]: W1103 23:04:12.921420 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243c55d2_1d9c_4e22_be54_9360d9f506f3.slice/crio-e5e4fe71f1f2df8ff56cb5fd11a4d1d3faf28b08d110df4e193b8eb8ddf65040 WatchSource:0}: Error finding container e5e4fe71f1f2df8ff56cb5fd11a4d1d3faf28b08d110df4e193b8eb8ddf65040: Status 404 returned error can't find the container with id e5e4fe71f1f2df8ff56cb5fd11a4d1d3faf28b08d110df4e193b8eb8ddf65040 Nov 03 23:04:12.974578 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:12.974539 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v5q4x"] Nov 03 23:04:13.017692 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:13.017653 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v5q4x" event={"ID":"243c55d2-1d9c-4e22-be54-9360d9f506f3","Type":"ContainerStarted","Data":"ab9422f00ade04b38bc5015a3bc3ae848f623297e1497073d61c6005d6cddfec"} Nov 03 23:04:13.017984 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:13.017739 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v5q4x" event={"ID":"243c55d2-1d9c-4e22-be54-9360d9f506f3","Type":"ContainerStarted","Data":"e5e4fe71f1f2df8ff56cb5fd11a4d1d3faf28b08d110df4e193b8eb8ddf65040"} Nov 03 23:04:13.017984 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:13.017789 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v5q4x" Nov 03 23:04:13.187852 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:13.187743 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v5q4x" podStartSLOduration=1.1877287380000001 podStartE2EDuration="1.187728738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:13.136231926 +0000 UTC m=+3420.730263900" watchObservedRunningTime="2026-11-03 23:04:13.187728738 +0000 UTC m=+3420.781760713" Nov 03 23:04:24.028759 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:24.028717 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-v5q4x" Nov 03 23:04:42.022198 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:04:42.022154 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:04:57.216329 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:57.216204 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-l96p9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:04:57.217288 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:04:57.217229 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-l96p9" podUID="d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:05:12.024438 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:05:12.024399 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:05:15.047351 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:15.047306 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs"] Nov 03 23:05:15.100374 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:15.100323 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vqtgs"] Nov 03 23:05:16.627215 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:16.627180 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff937d3a-685d-4e82-a425-50b82ba33369" path="/var/lib/kubelet/pods/ff937d3a-685d-4e82-a425-50b82ba33369/volumes" Nov 03 23:05:29.739620 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.739575 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns"] Nov 03 23:05:29.744586 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.744557 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:05:29.764743 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.764705 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6jbr\" (UniqueName: \"kubernetes.io/projected/ce0367be-37fc-40a9-b70a-6c4596ba25cf-kube-api-access-h6jbr\") pod \"maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns\" (UID: \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:05:29.764942 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.764901 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0367be-37fc-40a9-b70a-6c4596ba25cf-metrics-certs\") pod \"maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns\" (UID: \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:05:29.765005 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.764950 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0367be-37fc-40a9-b70a-6c4596ba25cf-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns\" (UID: \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:05:29.765179 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.765154 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0367be-37fc-40a9-b70a-6c4596ba25cf-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns\" (UID: \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:05:29.804027 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.803985 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-28c170-serving-cert\"" Nov 03 23:05:29.854774 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.854739 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vjv49\"" Nov 03 23:05:29.854774 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.854749 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 23:05:29.854774 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.854771 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:05:29.854774 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.854786 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:05:29.857848 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.857815 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns"] Nov 03 23:05:29.866383 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.866346 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jbr\" (UniqueName: \"kubernetes.io/projected/ce0367be-37fc-40a9-b70a-6c4596ba25cf-kube-api-access-h6jbr\") pod \"maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns\" (UID: \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:05:29.866578 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.866391 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0367be-37fc-40a9-b70a-6c4596ba25cf-metrics-certs\") pod \"maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns\" (UID: \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:05:29.866578 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.866417 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0367be-37fc-40a9-b70a-6c4596ba25cf-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns\" (UID: \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:05:29.866578 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.866470 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0367be-37fc-40a9-b70a-6c4596ba25cf-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns\" (UID: \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:05:29.866788 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:05:29.866569 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-serving-cert: secret "maas-api-e2e-worker-single-main-28c170-serving-cert" not found Nov 03 23:05:29.866788 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:05:29.866660 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0367be-37fc-40a9-b70a-6c4596ba25cf-maas-api-tls podName:ce0367be-37fc-40a9-b70a-6c4596ba25cf nodeName:}" failed. No retries permitted until 2026-11-03 23:05:30.366638406 +0000 UTC m=+3497.960670379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce0367be-37fc-40a9-b70a-6c4596ba25cf-maas-api-tls") pod "maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" (UID: "ce0367be-37fc-40a9-b70a-6c4596ba25cf") : secret "maas-api-e2e-worker-single-main-28c170-serving-cert" not found Nov 03 23:05:29.867286 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.867266 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0367be-37fc-40a9-b70a-6c4596ba25cf-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns\" (UID: \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:05:29.868940 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.868914 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0367be-37fc-40a9-b70a-6c4596ba25cf-metrics-certs\") pod \"maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns\" (UID: \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:05:29.924848 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:29.924810 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6jbr\" (UniqueName: \"kubernetes.io/projected/ce0367be-37fc-40a9-b70a-6c4596ba25cf-kube-api-access-h6jbr\") pod \"maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns\" (UID: \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:05:30.378797 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:30.378758 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0367be-37fc-40a9-b70a-6c4596ba25cf-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns\" (UID: \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:05:30.381441 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:30.381403 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0367be-37fc-40a9-b70a-6c4596ba25cf-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns\" (UID: \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:05:30.658603 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:30.658496 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:05:31.038945 ip-10-0-141-84 kubenswrapper[5084]: W1103 23:05:31.038909 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0367be_37fc_40a9_b70a_6c4596ba25cf.slice/crio-6ed50d5c6282744e4989e2af403037a1158d8dcb1d784981d92562148ddcac9e WatchSource:0}: Error finding container 6ed50d5c6282744e4989e2af403037a1158d8dcb1d784981d92562148ddcac9e: Status 404 returned error can't find the container with id 6ed50d5c6282744e4989e2af403037a1158d8dcb1d784981d92562148ddcac9e Nov 03 23:05:31.041450 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:31.041419 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns"] Nov 03 23:05:31.504968 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:31.504878 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" event={"ID":"ce0367be-37fc-40a9-b70a-6c4596ba25cf","Type":"ContainerStarted","Data":"6ed50d5c6282744e4989e2af403037a1158d8dcb1d784981d92562148ddcac9e"} Nov 03 23:05:32.512192 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:32.512075 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" event={"ID":"ce0367be-37fc-40a9-b70a-6c4596ba25cf","Type":"ContainerStarted","Data":"54ad2ecb2623aebb4f5133864a6cb871b1ef7d3a90f3c1d371671d6d739cfb61"} Nov 03 23:05:32.512192 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:32.512156 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:05:32.638299 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:32.638232 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" podStartSLOduration=2.642052866 podStartE2EDuration="3.638210106s" totalImagesPullingTime="996.15724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:05:32.634809534 +0000 UTC m=+3500.228841502" watchObservedRunningTime="2026-11-03 23:05:32.638210106 +0000 UTC m=+3500.232242081" Nov 03 23:05:38.524021 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:05:38.523988 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:05:42.022139 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:05:42.022101 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:06:12.025360 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:06:12.025318 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:06:17.422739 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:06:17.422705 5084 scope.go:122] "RemoveContainer" containerID="e047655352623685837b25076a154336c5cb6954608893f595644d77461685cd" Nov 03 23:06:42.023129 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:06:42.023082 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:06:59.314535 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:06:59.314490 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns"] Nov 03 23:06:59.315190 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:06:59.314735 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" podUID="ce0367be-37fc-40a9-b70a-6c4596ba25cf" containerName="maas-api" containerID="cri-o://54ad2ecb2623aebb4f5133864a6cb871b1ef7d3a90f3c1d371671d6d739cfb61" gracePeriod=30 Nov 03 23:07:00.580984 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:00.580947 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:07:00.622712 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:00.622677 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce0367be-37fc-40a9-b70a-6c4596ba25cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0367be-37fc-40a9-b70a-6c4596ba25cf-maas-api-tls\") pod \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\" (UID: \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\") " Nov 03 23:07:00.622922 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:00.622794 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce0367be-37fc-40a9-b70a-6c4596ba25cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0367be-37fc-40a9-b70a-6c4596ba25cf-openshift-service-ca\") pod \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\" (UID: \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\") " Nov 03 23:07:00.622922 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:00.622817 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce0367be-37fc-40a9-b70a-6c4596ba25cf-kube-api-access-h6jbr\" (UniqueName: \"kubernetes.io/projected/ce0367be-37fc-40a9-b70a-6c4596ba25cf-kube-api-access-h6jbr\") pod \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\" (UID: \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\") " Nov 03 23:07:00.622922 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:00.622872 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce0367be-37fc-40a9-b70a-6c4596ba25cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0367be-37fc-40a9-b70a-6c4596ba25cf-metrics-certs\") pod \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\" (UID: \"ce0367be-37fc-40a9-b70a-6c4596ba25cf\") " Nov 03 23:07:00.623310 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:00.623277 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0367be-37fc-40a9-b70a-6c4596ba25cf-openshift-service-ca" pod "ce0367be-37fc-40a9-b70a-6c4596ba25cf" (UID: "ce0367be-37fc-40a9-b70a-6c4596ba25cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:00.625207 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:00.625136 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce0367be-37fc-40a9-b70a-6c4596ba25cf-maas-api-tls" pod "ce0367be-37fc-40a9-b70a-6c4596ba25cf" (UID: "ce0367be-37fc-40a9-b70a-6c4596ba25cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:00.625312 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:00.625267 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce0367be-37fc-40a9-b70a-6c4596ba25cf-metrics-certs" pod "ce0367be-37fc-40a9-b70a-6c4596ba25cf" (UID: "ce0367be-37fc-40a9-b70a-6c4596ba25cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:00.625565 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:00.625535 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0367be-37fc-40a9-b70a-6c4596ba25cf-kube-api-access-h6jbr" pod "ce0367be-37fc-40a9-b70a-6c4596ba25cf" (UID: "ce0367be-37fc-40a9-b70a-6c4596ba25cf"). InnerVolumeSpecName "kube-api-access-h6jbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:00.724648 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:00.724599 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0367be-37fc-40a9-b70a-6c4596ba25cf-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:00.724648 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:00.724635 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0367be-37fc-40a9-b70a-6c4596ba25cf-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:00.724648 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:00.724645 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6jbr\" (UniqueName: \"kubernetes.io/projected/ce0367be-37fc-40a9-b70a-6c4596ba25cf-kube-api-access-h6jbr\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:00.724648 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:00.724657 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0367be-37fc-40a9-b70a-6c4596ba25cf-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:07:01.035264 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:01.035223 5084 generic.go:356] "Generic (PLEG): container finished" podID="ce0367be-37fc-40a9-b70a-6c4596ba25cf" containerID="54ad2ecb2623aebb4f5133864a6cb871b1ef7d3a90f3c1d371671d6d739cfb61" exitCode=0 Nov 03 23:07:01.035264 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:01.035268 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" Nov 03 23:07:01.035510 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:01.035305 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" event={"ID":"ce0367be-37fc-40a9-b70a-6c4596ba25cf","Type":"ContainerDied","Data":"54ad2ecb2623aebb4f5133864a6cb871b1ef7d3a90f3c1d371671d6d739cfb61"} Nov 03 23:07:01.035510 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:01.035363 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns" event={"ID":"ce0367be-37fc-40a9-b70a-6c4596ba25cf","Type":"ContainerDied","Data":"6ed50d5c6282744e4989e2af403037a1158d8dcb1d784981d92562148ddcac9e"} Nov 03 23:07:01.035510 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:01.035385 5084 scope.go:122] "RemoveContainer" containerID="54ad2ecb2623aebb4f5133864a6cb871b1ef7d3a90f3c1d371671d6d739cfb61" Nov 03 23:07:01.045616 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:01.045585 5084 scope.go:122] "RemoveContainer" containerID="54ad2ecb2623aebb4f5133864a6cb871b1ef7d3a90f3c1d371671d6d739cfb61" Nov 03 23:07:01.045984 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:07:01.045955 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ad2ecb2623aebb4f5133864a6cb871b1ef7d3a90f3c1d371671d6d739cfb61\": container with ID starting with 54ad2ecb2623aebb4f5133864a6cb871b1ef7d3a90f3c1d371671d6d739cfb61 not found: ID does not exist" containerID="54ad2ecb2623aebb4f5133864a6cb871b1ef7d3a90f3c1d371671d6d739cfb61" Nov 03 23:07:01.046123 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:01.045991 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ad2ecb2623aebb4f5133864a6cb871b1ef7d3a90f3c1d371671d6d739cfb61"} err="failed to get container status \"54ad2ecb2623aebb4f5133864a6cb871b1ef7d3a90f3c1d371671d6d739cfb61\": rpc error: code = NotFound desc = could not find container \"54ad2ecb2623aebb4f5133864a6cb871b1ef7d3a90f3c1d371671d6d739cfb61\": container with ID starting with 54ad2ecb2623aebb4f5133864a6cb871b1ef7d3a90f3c1d371671d6d739cfb61 not found: ID does not exist" Nov 03 23:07:01.203915 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:01.203867 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns"] Nov 03 23:07:01.207126 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:01.207096 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-28c170-54c4578447-bvpns"] Nov 03 23:07:02.627701 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:02.627665 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce0367be-37fc-40a9-b70a-6c4596ba25cf" path="/var/lib/kubelet/pods/ce0367be-37fc-40a9-b70a-6c4596ba25cf/volumes" Nov 03 23:07:12.023808 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:07:12.023770 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:07:13.418113 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:13.418080 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 23:07:13.421610 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:13.421580 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 23:07:26.314269 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:26.314213 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:07:26.314903 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:07:26.314400 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-kgjwz" podUID="8cebf3c1-c5fb-4feb-beac-7cf2e8a01134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:07:42.022695 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:07:42.022646 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:08:12.025264 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:08:12.025214 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:08:40.036332 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:08:40.036274 5084 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d64fcddb7-28lxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:08:40.036975 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:08:40.036357 5084 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d64fcddb7-28lxj" podUID="d3d5085e-0b4b-46a0-83cc-339d5550dd95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:08:42.023132 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:08:42.023092 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:09:12.024534 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:09:12.024494 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:09:42.024740 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:09:42.024686 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:09:42.411141 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.411035 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz"] Nov 03 23:09:42.411518 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.411502 5084 cpu_manager.go:414] "containerMap: removing container" podUID="ce0367be-37fc-40a9-b70a-6c4596ba25cf" containerName="maas-api" Nov 03 23:09:42.411563 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.411520 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce0367be-37fc-40a9-b70a-6c4596ba25cf" containerName="maas-api" Nov 03 23:09:42.411640 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.411631 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce0367be-37fc-40a9-b70a-6c4596ba25cf" containerName="maas-api" Nov 03 23:09:42.415020 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.414997 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" Nov 03 23:09:42.467988 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.467947 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz"] Nov 03 23:09:42.516422 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.516371 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:09:42.516619 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.516489 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:09:42.519838 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.519808 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 23:09:42.519997 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.519841 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vjv49\"" Nov 03 23:09:42.519997 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.519866 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 23:09:42.525525 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.525486 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c0564a-8d70-498b-b3d4-f26b1a027001-openshift-service-ca\") pod \"maas-api-bb597cf6f-z8ldz\" (UID: \"a0c0564a-8d70-498b-b3d4-f26b1a027001\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" Nov 03 23:09:42.525642 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.525532 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c0564a-8d70-498b-b3d4-f26b1a027001-maas-api-tls\") pod \"maas-api-bb597cf6f-z8ldz\" (UID: \"a0c0564a-8d70-498b-b3d4-f26b1a027001\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" Nov 03 23:09:42.525686 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.525652 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c0564a-8d70-498b-b3d4-f26b1a027001-metrics-certs\") pod \"maas-api-bb597cf6f-z8ldz\" (UID: \"a0c0564a-8d70-498b-b3d4-f26b1a027001\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" Nov 03 23:09:42.525686 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.525680 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzrjn\" (UniqueName: \"kubernetes.io/projected/a0c0564a-8d70-498b-b3d4-f26b1a027001-kube-api-access-wzrjn\") pod \"maas-api-bb597cf6f-z8ldz\" (UID: \"a0c0564a-8d70-498b-b3d4-f26b1a027001\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" Nov 03 23:09:42.626541 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.626503 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c0564a-8d70-498b-b3d4-f26b1a027001-metrics-certs\") pod \"maas-api-bb597cf6f-z8ldz\" (UID: \"a0c0564a-8d70-498b-b3d4-f26b1a027001\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" Nov 03 23:09:42.626541 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.626540 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzrjn\" (UniqueName: \"kubernetes.io/projected/a0c0564a-8d70-498b-b3d4-f26b1a027001-kube-api-access-wzrjn\") pod \"maas-api-bb597cf6f-z8ldz\" (UID: \"a0c0564a-8d70-498b-b3d4-f26b1a027001\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" Nov 03 23:09:42.626817 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.626605 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c0564a-8d70-498b-b3d4-f26b1a027001-openshift-service-ca\") pod \"maas-api-bb597cf6f-z8ldz\" (UID: \"a0c0564a-8d70-498b-b3d4-f26b1a027001\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" Nov 03 23:09:42.626817 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.626621 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c0564a-8d70-498b-b3d4-f26b1a027001-maas-api-tls\") pod \"maas-api-bb597cf6f-z8ldz\" (UID: \"a0c0564a-8d70-498b-b3d4-f26b1a027001\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" Nov 03 23:09:42.627330 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.627304 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c0564a-8d70-498b-b3d4-f26b1a027001-openshift-service-ca\") pod \"maas-api-bb597cf6f-z8ldz\" (UID: \"a0c0564a-8d70-498b-b3d4-f26b1a027001\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" Nov 03 23:09:42.629191 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.629164 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c0564a-8d70-498b-b3d4-f26b1a027001-maas-api-tls\") pod \"maas-api-bb597cf6f-z8ldz\" (UID: \"a0c0564a-8d70-498b-b3d4-f26b1a027001\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" Nov 03 23:09:42.629305 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.629218 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c0564a-8d70-498b-b3d4-f26b1a027001-metrics-certs\") pod \"maas-api-bb597cf6f-z8ldz\" (UID: \"a0c0564a-8d70-498b-b3d4-f26b1a027001\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" Nov 03 23:09:42.685144 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.685004 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzrjn\" (UniqueName: \"kubernetes.io/projected/a0c0564a-8d70-498b-b3d4-f26b1a027001-kube-api-access-wzrjn\") pod \"maas-api-bb597cf6f-z8ldz\" (UID: \"a0c0564a-8d70-498b-b3d4-f26b1a027001\") " pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" Nov 03 23:09:42.728971 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:42.728923 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" Nov 03 23:09:43.102600 ip-10-0-141-84 kubenswrapper[5084]: W1103 23:09:43.102569 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c0564a_8d70_498b_b3d4_f26b1a027001.slice/crio-f6fc74ab4d6032d16a5d1e9fabd722b4723453ac8eae49eb24cf947b0c54b5f1 WatchSource:0}: Error finding container f6fc74ab4d6032d16a5d1e9fabd722b4723453ac8eae49eb24cf947b0c54b5f1: Status 404 returned error can't find the container with id f6fc74ab4d6032d16a5d1e9fabd722b4723453ac8eae49eb24cf947b0c54b5f1 Nov 03 23:09:43.103997 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:43.103975 5084 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:09:43.105054 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:43.105029 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz"] Nov 03 23:09:43.241299 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:43.241253 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" event={"ID":"a0c0564a-8d70-498b-b3d4-f26b1a027001","Type":"ContainerStarted","Data":"f6fc74ab4d6032d16a5d1e9fabd722b4723453ac8eae49eb24cf947b0c54b5f1"} Nov 03 23:09:44.250389 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:44.250350 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" event={"ID":"a0c0564a-8d70-498b-b3d4-f26b1a027001","Type":"ContainerStarted","Data":"39372396c753014f31d110768b9fbab842c6bcead8d6dc1b65f2d8985ad15912"} Nov 03 23:09:44.250906 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:44.250450 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" Nov 03 23:09:44.369351 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:44.369238 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" podStartSLOduration=1.42357639 podStartE2EDuration="2.369224479s" totalImagesPullingTime="945.648089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:09:44.367286202 +0000 UTC m=+3751.961318176" watchObservedRunningTime="2026-11-03 23:09:44.369224479 +0000 UTC m=+3751.963256454" Nov 03 23:09:50.266844 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:50.266806 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" Nov 03 23:09:50.915004 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:50.914959 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79bf796845-8mlkm"] Nov 03 23:09:50.919632 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:50.919605 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" Nov 03 23:09:51.012515 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:51.012470 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb6c979a-84fa-4ec4-85c9-ff9f4389af5d-maas-api-tls\") pod \"maas-api-79bf796845-8mlkm\" (UID: \"bb6c979a-84fa-4ec4-85c9-ff9f4389af5d\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" Nov 03 23:09:51.012716 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:51.012534 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6c979a-84fa-4ec4-85c9-ff9f4389af5d-openshift-service-ca\") pod \"maas-api-79bf796845-8mlkm\" (UID: \"bb6c979a-84fa-4ec4-85c9-ff9f4389af5d\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" Nov 03 23:09:51.012716 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:51.012647 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glfgq\" (UniqueName: \"kubernetes.io/projected/bb6c979a-84fa-4ec4-85c9-ff9f4389af5d-kube-api-access-glfgq\") pod \"maas-api-79bf796845-8mlkm\" (UID: \"bb6c979a-84fa-4ec4-85c9-ff9f4389af5d\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" Nov 03 23:09:51.012843 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:51.012815 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6c979a-84fa-4ec4-85c9-ff9f4389af5d-metrics-certs\") pod \"maas-api-79bf796845-8mlkm\" (UID: \"bb6c979a-84fa-4ec4-85c9-ff9f4389af5d\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" Nov 03 23:09:51.026266 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:51.026221 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79bf796845-8mlkm"] Nov 03 23:09:51.113846 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:51.113797 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6c979a-84fa-4ec4-85c9-ff9f4389af5d-metrics-certs\") pod \"maas-api-79bf796845-8mlkm\" (UID: \"bb6c979a-84fa-4ec4-85c9-ff9f4389af5d\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" Nov 03 23:09:51.114044 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:51.113869 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb6c979a-84fa-4ec4-85c9-ff9f4389af5d-maas-api-tls\") pod \"maas-api-79bf796845-8mlkm\" (UID: \"bb6c979a-84fa-4ec4-85c9-ff9f4389af5d\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" Nov 03 23:09:51.114044 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:51.113899 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6c979a-84fa-4ec4-85c9-ff9f4389af5d-openshift-service-ca\") pod \"maas-api-79bf796845-8mlkm\" (UID: \"bb6c979a-84fa-4ec4-85c9-ff9f4389af5d\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" Nov 03 23:09:51.114044 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:51.113949 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glfgq\" (UniqueName: \"kubernetes.io/projected/bb6c979a-84fa-4ec4-85c9-ff9f4389af5d-kube-api-access-glfgq\") pod \"maas-api-79bf796845-8mlkm\" (UID: \"bb6c979a-84fa-4ec4-85c9-ff9f4389af5d\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" Nov 03 23:09:51.114665 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:51.114637 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6c979a-84fa-4ec4-85c9-ff9f4389af5d-openshift-service-ca\") pod \"maas-api-79bf796845-8mlkm\" (UID: \"bb6c979a-84fa-4ec4-85c9-ff9f4389af5d\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" Nov 03 23:09:51.116536 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:51.116503 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6c979a-84fa-4ec4-85c9-ff9f4389af5d-metrics-certs\") pod \"maas-api-79bf796845-8mlkm\" (UID: \"bb6c979a-84fa-4ec4-85c9-ff9f4389af5d\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" Nov 03 23:09:51.116536 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:51.116524 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb6c979a-84fa-4ec4-85c9-ff9f4389af5d-maas-api-tls\") pod \"maas-api-79bf796845-8mlkm\" (UID: \"bb6c979a-84fa-4ec4-85c9-ff9f4389af5d\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" Nov 03 23:09:51.171960 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:51.171876 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glfgq\" (UniqueName: \"kubernetes.io/projected/bb6c979a-84fa-4ec4-85c9-ff9f4389af5d-kube-api-access-glfgq\") pod \"maas-api-79bf796845-8mlkm\" (UID: \"bb6c979a-84fa-4ec4-85c9-ff9f4389af5d\") " pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" Nov 03 23:09:51.236667 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:51.236629 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" Nov 03 23:09:51.608308 ip-10-0-141-84 kubenswrapper[5084]: W1103 23:09:51.608273 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6c979a_84fa_4ec4_85c9_ff9f4389af5d.slice/crio-ec101a0095eea3c82f215b1eb4b1b88733c416159e226262eab509170dcf1677 WatchSource:0}: Error finding container ec101a0095eea3c82f215b1eb4b1b88733c416159e226262eab509170dcf1677: Status 404 returned error can't find the container with id ec101a0095eea3c82f215b1eb4b1b88733c416159e226262eab509170dcf1677 Nov 03 23:09:51.610807 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:51.610785 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79bf796845-8mlkm"] Nov 03 23:09:52.316771 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:52.316734 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" event={"ID":"bb6c979a-84fa-4ec4-85c9-ff9f4389af5d","Type":"ContainerStarted","Data":"ec101a0095eea3c82f215b1eb4b1b88733c416159e226262eab509170dcf1677"} Nov 03 23:09:53.325281 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:53.325227 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" event={"ID":"bb6c979a-84fa-4ec4-85c9-ff9f4389af5d","Type":"ContainerStarted","Data":"bd31bb1e68bb9d2d84e2f156d9ce265a3e70ed770d69a8923fbd67877387865c"} Nov 03 23:09:53.325856 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:53.325313 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" Nov 03 23:09:53.443494 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:53.443435 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" podStartSLOduration=2.463738833 podStartE2EDuration="3.443421637s" totalImagesPullingTime="979.682804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:09:53.442351149 +0000 UTC m=+3761.036383116" watchObservedRunningTime="2026-11-03 23:09:53.443421637 +0000 UTC m=+3761.037453610" Nov 03 23:09:59.336974 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:59.336938 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79bf796845-8mlkm" Nov 03 23:09:59.508874 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:59.508838 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz"] Nov 03 23:09:59.509137 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:59.509045 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" podUID="a0c0564a-8d70-498b-b3d4-f26b1a027001" containerName="maas-api" containerID="cri-o://39372396c753014f31d110768b9fbab842c6bcead8d6dc1b65f2d8985ad15912" gracePeriod=30 Nov 03 23:09:59.782124 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:59.782089 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" Nov 03 23:09:59.911395 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:59.911300 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c0564a-8d70-498b-b3d4-f26b1a027001-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c0564a-8d70-498b-b3d4-f26b1a027001-openshift-service-ca\") pod \"a0c0564a-8d70-498b-b3d4-f26b1a027001\" (UID: \"a0c0564a-8d70-498b-b3d4-f26b1a027001\") " Nov 03 23:09:59.911395 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:59.911389 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c0564a-8d70-498b-b3d4-f26b1a027001-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c0564a-8d70-498b-b3d4-f26b1a027001-maas-api-tls\") pod \"a0c0564a-8d70-498b-b3d4-f26b1a027001\" (UID: \"a0c0564a-8d70-498b-b3d4-f26b1a027001\") " Nov 03 23:09:59.911641 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:59.911416 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c0564a-8d70-498b-b3d4-f26b1a027001-kube-api-access-wzrjn\" (UniqueName: \"kubernetes.io/projected/a0c0564a-8d70-498b-b3d4-f26b1a027001-kube-api-access-wzrjn\") pod \"a0c0564a-8d70-498b-b3d4-f26b1a027001\" (UID: \"a0c0564a-8d70-498b-b3d4-f26b1a027001\") " Nov 03 23:09:59.911641 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:59.911477 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c0564a-8d70-498b-b3d4-f26b1a027001-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c0564a-8d70-498b-b3d4-f26b1a027001-metrics-certs\") pod \"a0c0564a-8d70-498b-b3d4-f26b1a027001\" (UID: \"a0c0564a-8d70-498b-b3d4-f26b1a027001\") " Nov 03 23:09:59.911765 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:59.911731 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c0564a-8d70-498b-b3d4-f26b1a027001-openshift-service-ca" pod "a0c0564a-8d70-498b-b3d4-f26b1a027001" (UID: "a0c0564a-8d70-498b-b3d4-f26b1a027001"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:09:59.911956 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:59.911929 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c0564a-8d70-498b-b3d4-f26b1a027001-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:09:59.913869 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:59.913836 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c0564a-8d70-498b-b3d4-f26b1a027001-kube-api-access-wzrjn" pod "a0c0564a-8d70-498b-b3d4-f26b1a027001" (UID: "a0c0564a-8d70-498b-b3d4-f26b1a027001"). InnerVolumeSpecName "kube-api-access-wzrjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:09:59.914210 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:59.914173 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c0564a-8d70-498b-b3d4-f26b1a027001-maas-api-tls" pod "a0c0564a-8d70-498b-b3d4-f26b1a027001" (UID: "a0c0564a-8d70-498b-b3d4-f26b1a027001"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:09:59.914210 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:09:59.914187 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c0564a-8d70-498b-b3d4-f26b1a027001-metrics-certs" pod "a0c0564a-8d70-498b-b3d4-f26b1a027001" (UID: "a0c0564a-8d70-498b-b3d4-f26b1a027001"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:10:00.013607 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:10:00.013567 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c0564a-8d70-498b-b3d4-f26b1a027001-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:00.013607 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:10:00.013606 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzrjn\" (UniqueName: \"kubernetes.io/projected/a0c0564a-8d70-498b-b3d4-f26b1a027001-kube-api-access-wzrjn\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:00.013839 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:10:00.013617 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c0564a-8d70-498b-b3d4-f26b1a027001-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:10:00.371173 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:10:00.371137 5084 generic.go:356] "Generic (PLEG): container finished" podID="a0c0564a-8d70-498b-b3d4-f26b1a027001" containerID="39372396c753014f31d110768b9fbab842c6bcead8d6dc1b65f2d8985ad15912" exitCode=0 Nov 03 23:10:00.371647 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:10:00.371181 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" Nov 03 23:10:00.371647 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:10:00.371225 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" event={"ID":"a0c0564a-8d70-498b-b3d4-f26b1a027001","Type":"ContainerDied","Data":"39372396c753014f31d110768b9fbab842c6bcead8d6dc1b65f2d8985ad15912"} Nov 03 23:10:00.371647 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:10:00.371289 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz" event={"ID":"a0c0564a-8d70-498b-b3d4-f26b1a027001","Type":"ContainerDied","Data":"f6fc74ab4d6032d16a5d1e9fabd722b4723453ac8eae49eb24cf947b0c54b5f1"} Nov 03 23:10:00.371647 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:10:00.371314 5084 scope.go:122] "RemoveContainer" containerID="39372396c753014f31d110768b9fbab842c6bcead8d6dc1b65f2d8985ad15912" Nov 03 23:10:00.382210 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:10:00.382185 5084 scope.go:122] "RemoveContainer" containerID="39372396c753014f31d110768b9fbab842c6bcead8d6dc1b65f2d8985ad15912" Nov 03 23:10:00.382588 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:10:00.382563 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39372396c753014f31d110768b9fbab842c6bcead8d6dc1b65f2d8985ad15912\": container with ID starting with 39372396c753014f31d110768b9fbab842c6bcead8d6dc1b65f2d8985ad15912 not found: ID does not exist" containerID="39372396c753014f31d110768b9fbab842c6bcead8d6dc1b65f2d8985ad15912" Nov 03 23:10:00.382696 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:10:00.382594 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39372396c753014f31d110768b9fbab842c6bcead8d6dc1b65f2d8985ad15912"} err="failed to get container status \"39372396c753014f31d110768b9fbab842c6bcead8d6dc1b65f2d8985ad15912\": rpc error: code = NotFound desc = could not find container \"39372396c753014f31d110768b9fbab842c6bcead8d6dc1b65f2d8985ad15912\": container with ID starting with 39372396c753014f31d110768b9fbab842c6bcead8d6dc1b65f2d8985ad15912 not found: ID does not exist" Nov 03 23:10:00.538789 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:10:00.538750 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz"] Nov 03 23:10:00.589693 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:10:00.589657 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-bb597cf6f-z8ldz"] Nov 03 23:10:00.627576 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:10:00.627497 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c0564a-8d70-498b-b3d4-f26b1a027001" path="/var/lib/kubelet/pods/a0c0564a-8d70-498b-b3d4-f26b1a027001/volumes" Nov 03 23:10:12.024860 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:10:12.024817 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:10:42.028823 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:10:42.028779 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:11:12.025211 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:11:12.025168 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:11:33.621651 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:33.621609 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9"] Nov 03 23:11:33.622119 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:33.622090 5084 cpu_manager.go:414] "containerMap: removing container" podUID="a0c0564a-8d70-498b-b3d4-f26b1a027001" containerName="maas-api" Nov 03 23:11:33.622119 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:33.622104 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c0564a-8d70-498b-b3d4-f26b1a027001" containerName="maas-api" Nov 03 23:11:33.622205 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:33.622192 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c0564a-8d70-498b-b3d4-f26b1a027001" containerName="maas-api" Nov 03 23:11:33.625633 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:33.625611 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:11:33.680739 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:33.680677 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-26c7c9-serving-cert\"" Nov 03 23:11:33.734542 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:33.734497 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9"] Nov 03 23:11:33.780947 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:33.780906 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1683ab11-dadf-432a-abe3-a7af12d46f82-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9\" (UID: \"1683ab11-dadf-432a-abe3-a7af12d46f82\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:11:33.781172 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:33.781126 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1683ab11-dadf-432a-abe3-a7af12d46f82-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9\" (UID: \"1683ab11-dadf-432a-abe3-a7af12d46f82\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:11:33.781254 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:33.781172 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sws6z\" (UniqueName: \"kubernetes.io/projected/1683ab11-dadf-432a-abe3-a7af12d46f82-kube-api-access-sws6z\") pod \"maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9\" (UID: \"1683ab11-dadf-432a-abe3-a7af12d46f82\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:11:33.781322 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:33.781249 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1683ab11-dadf-432a-abe3-a7af12d46f82-metrics-certs\") pod \"maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9\" (UID: \"1683ab11-dadf-432a-abe3-a7af12d46f82\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:11:33.882967 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:33.882292 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1683ab11-dadf-432a-abe3-a7af12d46f82-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9\" (UID: \"1683ab11-dadf-432a-abe3-a7af12d46f82\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:11:33.882967 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:33.882358 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sws6z\" (UniqueName: \"kubernetes.io/projected/1683ab11-dadf-432a-abe3-a7af12d46f82-kube-api-access-sws6z\") pod \"maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9\" (UID: \"1683ab11-dadf-432a-abe3-a7af12d46f82\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:11:33.882967 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:33.882402 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1683ab11-dadf-432a-abe3-a7af12d46f82-metrics-certs\") pod \"maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9\" (UID: \"1683ab11-dadf-432a-abe3-a7af12d46f82\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:11:33.882967 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:33.882482 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1683ab11-dadf-432a-abe3-a7af12d46f82-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9\" (UID: \"1683ab11-dadf-432a-abe3-a7af12d46f82\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:11:33.886512 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:11:33.883930 5084 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-serving-cert: secret "maas-api-e2e-authcfg-http-26c7c9-serving-cert" not found Nov 03 23:11:33.886512 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:11:33.884018 5084 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1683ab11-dadf-432a-abe3-a7af12d46f82-maas-api-tls podName:1683ab11-dadf-432a-abe3-a7af12d46f82 nodeName:}" failed. No retries permitted until 2026-11-03 23:11:34.38399797 +0000 UTC m=+3861.978029929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1683ab11-dadf-432a-abe3-a7af12d46f82-maas-api-tls") pod "maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" (UID: "1683ab11-dadf-432a-abe3-a7af12d46f82") : secret "maas-api-e2e-authcfg-http-26c7c9-serving-cert" not found Nov 03 23:11:33.888459 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:33.888420 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1683ab11-dadf-432a-abe3-a7af12d46f82-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9\" (UID: \"1683ab11-dadf-432a-abe3-a7af12d46f82\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:11:33.892437 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:33.892343 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1683ab11-dadf-432a-abe3-a7af12d46f82-metrics-certs\") pod \"maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9\" (UID: \"1683ab11-dadf-432a-abe3-a7af12d46f82\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:11:33.947994 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:33.947956 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sws6z\" (UniqueName: \"kubernetes.io/projected/1683ab11-dadf-432a-abe3-a7af12d46f82-kube-api-access-sws6z\") pod \"maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9\" (UID: \"1683ab11-dadf-432a-abe3-a7af12d46f82\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:11:34.386908 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:34.386860 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1683ab11-dadf-432a-abe3-a7af12d46f82-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9\" (UID: \"1683ab11-dadf-432a-abe3-a7af12d46f82\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:11:34.389434 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:34.389398 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1683ab11-dadf-432a-abe3-a7af12d46f82-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9\" (UID: \"1683ab11-dadf-432a-abe3-a7af12d46f82\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:11:34.539822 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:34.539778 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:11:34.913346 ip-10-0-141-84 kubenswrapper[5084]: W1103 23:11:34.913317 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1683ab11_dadf_432a_abe3_a7af12d46f82.slice/crio-99743e9b3b57eb81c1894ee73179e9b300f375cc59f04b96a399c8d109809296 WatchSource:0}: Error finding container 99743e9b3b57eb81c1894ee73179e9b300f375cc59f04b96a399c8d109809296: Status 404 returned error can't find the container with id 99743e9b3b57eb81c1894ee73179e9b300f375cc59f04b96a399c8d109809296 Nov 03 23:11:34.916246 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:34.916223 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9"] Nov 03 23:11:34.945214 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:34.945176 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" event={"ID":"1683ab11-dadf-432a-abe3-a7af12d46f82","Type":"ContainerStarted","Data":"99743e9b3b57eb81c1894ee73179e9b300f375cc59f04b96a399c8d109809296"} Nov 03 23:11:35.953690 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:35.953638 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" event={"ID":"1683ab11-dadf-432a-abe3-a7af12d46f82","Type":"ContainerStarted","Data":"7c02b92c6ae89af6ff29509365cffade126a2307567f59afc3225faa46e5ab86"} Nov 03 23:11:35.954156 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:35.953721 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:11:36.113328 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:36.113219 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" podStartSLOduration=2.207741359 podStartE2EDuration="3.113203425s" totalImagesPullingTime="905.462066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:11:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:11:36.112306924 +0000 UTC m=+3863.706338898" watchObservedRunningTime="2026-11-03 23:11:36.113203425 +0000 UTC m=+3863.707235399" Nov 03 23:11:41.966309 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:11:41.966277 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:11:42.023661 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:11:42.023620 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:12:12.024780 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:12:12.024731 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:12:13.458381 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:12:13.458354 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 23:12:13.464994 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:12:13.464955 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 23:12:42.023004 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:12:42.022960 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:13:12.024806 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:13:12.024751 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:13:32.925521 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:32.925480 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9"] Nov 03 23:13:32.926057 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:32.925720 5084 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" podUID="1683ab11-dadf-432a-abe3-a7af12d46f82" containerName="maas-api" containerID="cri-o://7c02b92c6ae89af6ff29509365cffade126a2307567f59afc3225faa46e5ab86" gracePeriod=30 Nov 03 23:13:33.212516 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.212485 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:13:33.307035 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.306995 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1683ab11-dadf-432a-abe3-a7af12d46f82-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1683ab11-dadf-432a-abe3-a7af12d46f82-maas-api-tls\") pod \"1683ab11-dadf-432a-abe3-a7af12d46f82\" (UID: \"1683ab11-dadf-432a-abe3-a7af12d46f82\") " Nov 03 23:13:33.307328 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.307096 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1683ab11-dadf-432a-abe3-a7af12d46f82-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1683ab11-dadf-432a-abe3-a7af12d46f82-metrics-certs\") pod \"1683ab11-dadf-432a-abe3-a7af12d46f82\" (UID: \"1683ab11-dadf-432a-abe3-a7af12d46f82\") " Nov 03 23:13:33.307328 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.307146 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1683ab11-dadf-432a-abe3-a7af12d46f82-kube-api-access-sws6z\" (UniqueName: \"kubernetes.io/projected/1683ab11-dadf-432a-abe3-a7af12d46f82-kube-api-access-sws6z\") pod \"1683ab11-dadf-432a-abe3-a7af12d46f82\" (UID: \"1683ab11-dadf-432a-abe3-a7af12d46f82\") " Nov 03 23:13:33.307328 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.307241 5084 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1683ab11-dadf-432a-abe3-a7af12d46f82-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1683ab11-dadf-432a-abe3-a7af12d46f82-openshift-service-ca\") pod \"1683ab11-dadf-432a-abe3-a7af12d46f82\" (UID: \"1683ab11-dadf-432a-abe3-a7af12d46f82\") " Nov 03 23:13:33.308952 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.308915 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1683ab11-dadf-432a-abe3-a7af12d46f82-openshift-service-ca" pod "1683ab11-dadf-432a-abe3-a7af12d46f82" (UID: "1683ab11-dadf-432a-abe3-a7af12d46f82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:13:33.328197 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.327036 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1683ab11-dadf-432a-abe3-a7af12d46f82-metrics-certs" pod "1683ab11-dadf-432a-abe3-a7af12d46f82" (UID: "1683ab11-dadf-432a-abe3-a7af12d46f82"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:13:33.328197 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.327885 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1683ab11-dadf-432a-abe3-a7af12d46f82-maas-api-tls" pod "1683ab11-dadf-432a-abe3-a7af12d46f82" (UID: "1683ab11-dadf-432a-abe3-a7af12d46f82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:13:33.331190 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.330872 5084 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1683ab11-dadf-432a-abe3-a7af12d46f82-kube-api-access-sws6z" pod "1683ab11-dadf-432a-abe3-a7af12d46f82" (UID: "1683ab11-dadf-432a-abe3-a7af12d46f82"). InnerVolumeSpecName "kube-api-access-sws6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:13:33.409101 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.409027 5084 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1683ab11-dadf-432a-abe3-a7af12d46f82-openshift-service-ca\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:33.409297 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.409110 5084 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1683ab11-dadf-432a-abe3-a7af12d46f82-maas-api-tls\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:33.409297 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.409127 5084 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1683ab11-dadf-432a-abe3-a7af12d46f82-metrics-certs\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:33.409297 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.409159 5084 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sws6z\" (UniqueName: \"kubernetes.io/projected/1683ab11-dadf-432a-abe3-a7af12d46f82-kube-api-access-sws6z\") on node \"ip-10-0-141-84.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 23:13:33.682431 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.682398 5084 generic.go:356] "Generic (PLEG): container finished" podID="1683ab11-dadf-432a-abe3-a7af12d46f82" containerID="7c02b92c6ae89af6ff29509365cffade126a2307567f59afc3225faa46e5ab86" exitCode=0 Nov 03 23:13:33.682619 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.682453 5084 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" Nov 03 23:13:33.682619 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.682480 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" event={"ID":"1683ab11-dadf-432a-abe3-a7af12d46f82","Type":"ContainerDied","Data":"7c02b92c6ae89af6ff29509365cffade126a2307567f59afc3225faa46e5ab86"} Nov 03 23:13:33.682619 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.682539 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9" event={"ID":"1683ab11-dadf-432a-abe3-a7af12d46f82","Type":"ContainerDied","Data":"99743e9b3b57eb81c1894ee73179e9b300f375cc59f04b96a399c8d109809296"} Nov 03 23:13:33.682619 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.682563 5084 scope.go:122] "RemoveContainer" containerID="7c02b92c6ae89af6ff29509365cffade126a2307567f59afc3225faa46e5ab86" Nov 03 23:13:33.693695 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.693671 5084 scope.go:122] "RemoveContainer" containerID="7c02b92c6ae89af6ff29509365cffade126a2307567f59afc3225faa46e5ab86" Nov 03 23:13:33.694097 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:13:33.694043 5084 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c02b92c6ae89af6ff29509365cffade126a2307567f59afc3225faa46e5ab86\": container with ID starting with 7c02b92c6ae89af6ff29509365cffade126a2307567f59afc3225faa46e5ab86 not found: ID does not exist" containerID="7c02b92c6ae89af6ff29509365cffade126a2307567f59afc3225faa46e5ab86" Nov 03 23:13:33.694097 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.694090 5084 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c02b92c6ae89af6ff29509365cffade126a2307567f59afc3225faa46e5ab86"} err="failed to get container status \"7c02b92c6ae89af6ff29509365cffade126a2307567f59afc3225faa46e5ab86\": rpc error: code = NotFound desc = could not find container \"7c02b92c6ae89af6ff29509365cffade126a2307567f59afc3225faa46e5ab86\": container with ID starting with 7c02b92c6ae89af6ff29509365cffade126a2307567f59afc3225faa46e5ab86 not found: ID does not exist" Nov 03 23:13:33.849825 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.849787 5084 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9"] Nov 03 23:13:33.906235 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:33.906193 5084 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-26c7c9-787f8fbdf4-dt5n9"] Nov 03 23:13:34.626926 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:13:34.626890 5084 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1683ab11-dadf-432a-abe3-a7af12d46f82" path="/var/lib/kubelet/pods/1683ab11-dadf-432a-abe3-a7af12d46f82/volumes" Nov 03 23:13:42.022440 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:13:42.022392 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:14:12.022834 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:14:12.022796 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:14:42.022073 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:14:42.022016 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:15:06.143774 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:06.143737 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6bc9bf8bf6-78nd4_6f959a8d-48e1-4a43-8ef7-ff3bb056a7ff/manager/0.log" Nov 03 23:15:06.796591 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:06.796538 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dmvn7_6517eb66-69c7-49b0-a3c6-5256dba7034f/manager/0.log" Nov 03 23:15:07.586485 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:07.586440 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57d7bb69fb-gj9mx_a1870f1b-ca5c-4885-aecd-710452f04516/manager/0.log" Nov 03 23:15:08.197994 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:08.197940 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hhkg4_5117984d-dade-4802-a318-948f508c7658/manager/0.log" Nov 03 23:15:09.277850 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:09.277809 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79bf796845-8mlkm_bb6c979a-84fa-4ec4-85c9-ff9f4389af5d/maas-api/0.log" Nov 03 23:15:12.023837 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:15:12.023798 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:15:13.016107 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:13.016057 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-vjp47_5e81d876-d741-4044-8fdd-dde9acf179a3/manager/0.log" Nov 03 23:15:13.243190 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:13.243145 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5jsx7_5cdcb400-fe77-4a4d-84b8-cecb65fe5a61/manager/0.log" Nov 03 23:15:13.731037 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:13.731000 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-v5q4x_243c55d2-1d9c-4e22-be54-9360d9f506f3/manager/0.log" Nov 03 23:15:15.424964 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:15.424928 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wjlzz_3e14a353-5cfa-47ed-805f-ecb9efde013c/discovery/0.log" Nov 03 23:15:15.928022 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:15.927995 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dbbd9b584-gcs6s_5daf51f7-c907-41a7-ba73-1846f91f157a/kube-auth-proxy/0.log" Nov 03 23:15:16.144839 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:16.144801 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-28lxj_d3d5085e-0b4b-46a0-83cc-339d5550dd95/istio-proxy/0.log" Nov 03 23:15:17.515962 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:17.515892 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-kgjwz_8cebf3c1-c5fb-4feb-beac-7cf2e8a01134/istio-proxy/0.log" Nov 03 23:15:17.734799 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:17.734767 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-l96p9_d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2/istio-proxy/0.log" Nov 03 23:15:19.008332 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:19.008299 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-7q7t4_1dedc6fe-4409-4add-b490-1a3b2747520b/payload-pre-processing/0.log" Nov 03 23:15:19.454678 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:19.454645 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84d846b7f5-4qjqn_b537a9eb-5abf-4f6b-9a35-0f13c5211bf2/router/0.log" Nov 03 23:15:21.062839 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:21.062800 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-zhzjl_2a6d1214-9fc7-40a3-930b-1513e14ef268/main/0.log" Nov 03 23:15:21.515735 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:21.515706 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qb78n_bbdfd9d6-132a-423d-b96a-a0119bc8b122/main/0.log" Nov 03 23:15:31.888327 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:31.888235 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6bc9bf8bf6-78nd4_6f959a8d-48e1-4a43-8ef7-ff3bb056a7ff/manager/0.log" Nov 03 23:15:32.563464 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:32.563430 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dmvn7_6517eb66-69c7-49b0-a3c6-5256dba7034f/manager/0.log" Nov 03 23:15:33.312325 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:33.312285 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57d7bb69fb-gj9mx_a1870f1b-ca5c-4885-aecd-710452f04516/manager/0.log" Nov 03 23:15:33.859464 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:33.859423 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hhkg4_5117984d-dade-4802-a318-948f508c7658/manager/0.log" Nov 03 23:15:34.732852 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:34.732813 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79bf796845-8mlkm_bb6c979a-84fa-4ec4-85c9-ff9f4389af5d/maas-api/0.log" Nov 03 23:15:37.576322 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:37.576281 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-vjp47_5e81d876-d741-4044-8fdd-dde9acf179a3/manager/0.log" Nov 03 23:15:37.804553 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:37.804518 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5jsx7_5cdcb400-fe77-4a4d-84b8-cecb65fe5a61/manager/0.log" Nov 03 23:15:38.245547 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:38.245458 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-v5q4x_243c55d2-1d9c-4e22-be54-9360d9f506f3/manager/0.log" Nov 03 23:15:39.734163 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:39.734133 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wjlzz_3e14a353-5cfa-47ed-805f-ecb9efde013c/discovery/0.log" Nov 03 23:15:40.247571 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:40.247536 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dbbd9b584-gcs6s_5daf51f7-c907-41a7-ba73-1846f91f157a/kube-auth-proxy/0.log" Nov 03 23:15:40.476835 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:40.476805 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-28lxj_d3d5085e-0b4b-46a0-83cc-339d5550dd95/istio-proxy/0.log" Nov 03 23:15:41.792162 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:41.792126 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-kgjwz_8cebf3c1-c5fb-4feb-beac-7cf2e8a01134/istio-proxy/0.log" Nov 03 23:15:42.025164 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:15:42.022325 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:15:42.034872 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:42.034829 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d64fcddb7-l96p9_d75afbc1-4de4-4b68-ae28-e0a8c0d39fe2/istio-proxy/0.log" Nov 03 23:15:43.320614 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:43.320579 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-7q7t4_1dedc6fe-4409-4add-b490-1a3b2747520b/payload-pre-processing/0.log" Nov 03 23:15:43.784578 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:43.784548 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84d846b7f5-4qjqn_b537a9eb-5abf-4f6b-9a35-0f13c5211bf2/router/0.log" Nov 03 23:15:45.121865 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:45.121833 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-zhzjl_2a6d1214-9fc7-40a3-930b-1513e14ef268/main/0.log" Nov 03 23:15:45.570296 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:15:45.570260 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qb78n_bbdfd9d6-132a-423d-b96a-a0119bc8b122/main/0.log" Nov 03 23:16:12.024606 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:16:12.024567 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:16:13.741719 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:16:13.741678 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57d7bb69fb-gj9mx_a1870f1b-ca5c-4885-aecd-710452f04516/manager/0.log" Nov 03 23:16:14.302577 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:16:14.302540 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57d7bb69fb-gj9mx_a1870f1b-ca5c-4885-aecd-710452f04516/manager/0.log" Nov 03 23:16:42.021695 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:16:42.021650 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:16:51.650137 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:16:51.650100 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57d7bb69fb-gj9mx_a1870f1b-ca5c-4885-aecd-710452f04516/manager/0.log" Nov 03 23:16:52.191489 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:16:52.191445 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57d7bb69fb-gj9mx_a1870f1b-ca5c-4885-aecd-710452f04516/manager/0.log" Nov 03 23:17:03.375773 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:03.375728 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8wmmt_99d8e137-a970-441d-9800-d1ce1c8f1f94/global-pull-secret-syncer/0.log" Nov 03 23:17:04.643408 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:04.643374 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-f88sj_fe0af137-f9ce-4f18-8329-3f7df6a728bb/connection-checker/0.log" Nov 03 23:17:05.075884 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:05.075852 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-tc9n2_ec658817-ab95-4b41-a457-d067b9754e3e/connection-checker/0.log" Nov 03 23:17:05.512143 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:05.512034 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-z2q8s_508293cb-d657-4cfb-a67a-0155a44d6e47/connection-checker/0.log" Nov 03 23:17:06.608710 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:06.608672 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k4rnh_19a326ff-4502-4590-9c0d-194f8cf14abd/konnectivity-agent/0.log" Nov 03 23:17:08.343639 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:08.343602 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-84.us-west-2.compute.internal_a3ff6dbfd235d6871a32270ec8f4d659/haproxy/0.log" Nov 03 23:17:12.028800 ip-10-0-141-84 kubenswrapper[5084]: E1103 23:17:12.028757 5084 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vsnwk" containerName="extractor" Nov 03 23:17:13.026875 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.026839 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-vjp47_5e81d876-d741-4044-8fdd-dde9acf179a3/manager/0.log" Nov 03 23:17:13.439338 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.439255 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5jsx7_5cdcb400-fe77-4a4d-84b8-cecb65fe5a61/manager/0.log" Nov 03 23:17:13.498276 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.498244 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 23:17:13.506975 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.506934 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9mgrk_d1498363-2c98-4046-8feb-abb772a80149/console-operator/1.log" Nov 03 23:17:13.525930 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.525895 5084 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd"] Nov 03 23:17:13.526360 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.526347 5084 cpu_manager.go:414] "containerMap: removing container" podUID="1683ab11-dadf-432a-abe3-a7af12d46f82" containerName="maas-api" Nov 03 23:17:13.526411 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.526361 5084 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1683ab11-dadf-432a-abe3-a7af12d46f82" containerName="maas-api" Nov 03 23:17:13.526459 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.526450 5084 memory_manager.go:365] "RemoveStaleState removing state" podUID="1683ab11-dadf-432a-abe3-a7af12d46f82" containerName="maas-api" Nov 03 23:17:13.529666 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.529646 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:13.564700 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.564648 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30cd1176-6c6e-4eb7-bc09-6d356e5e04a1-podres\") pod \"perf-node-gather-daemonset-fblxd\" (UID: \"30cd1176-6c6e-4eb7-bc09-6d356e5e04a1\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:13.564927 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.564790 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh2hh\" (UniqueName: \"kubernetes.io/projected/30cd1176-6c6e-4eb7-bc09-6d356e5e04a1-kube-api-access-rh2hh\") pod \"perf-node-gather-daemonset-fblxd\" (UID: \"30cd1176-6c6e-4eb7-bc09-6d356e5e04a1\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:13.564927 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.564910 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30cd1176-6c6e-4eb7-bc09-6d356e5e04a1-sys\") pod \"perf-node-gather-daemonset-fblxd\" (UID: \"30cd1176-6c6e-4eb7-bc09-6d356e5e04a1\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:13.565048 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.564965 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30cd1176-6c6e-4eb7-bc09-6d356e5e04a1-lib-modules\") pod \"perf-node-gather-daemonset-fblxd\" (UID: \"30cd1176-6c6e-4eb7-bc09-6d356e5e04a1\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:13.565133 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.565073 5084 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30cd1176-6c6e-4eb7-bc09-6d356e5e04a1-proc\") pod \"perf-node-gather-daemonset-fblxd\" (UID: \"30cd1176-6c6e-4eb7-bc09-6d356e5e04a1\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:13.582857 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.582817 5084 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9px9d\"/\"default-dockercfg-546s9\"" Nov 03 23:17:13.634052 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.634005 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9px9d\"/\"openshift-service-ca.crt\"" Nov 03 23:17:13.634052 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.634029 5084 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9px9d\"/\"kube-root-ca.crt\"" Nov 03 23:17:13.636655 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.636627 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd"] Nov 03 23:17:13.665940 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.665901 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30cd1176-6c6e-4eb7-bc09-6d356e5e04a1-proc\") pod \"perf-node-gather-daemonset-fblxd\" (UID: \"30cd1176-6c6e-4eb7-bc09-6d356e5e04a1\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:13.666170 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.666006 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30cd1176-6c6e-4eb7-bc09-6d356e5e04a1-proc\") pod \"perf-node-gather-daemonset-fblxd\" (UID: \"30cd1176-6c6e-4eb7-bc09-6d356e5e04a1\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:13.666170 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.666117 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30cd1176-6c6e-4eb7-bc09-6d356e5e04a1-podres\") pod \"perf-node-gather-daemonset-fblxd\" (UID: \"30cd1176-6c6e-4eb7-bc09-6d356e5e04a1\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:13.666170 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.666170 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30cd1176-6c6e-4eb7-bc09-6d356e5e04a1-podres\") pod \"perf-node-gather-daemonset-fblxd\" (UID: \"30cd1176-6c6e-4eb7-bc09-6d356e5e04a1\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:13.666376 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.666237 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh2hh\" (UniqueName: \"kubernetes.io/projected/30cd1176-6c6e-4eb7-bc09-6d356e5e04a1-kube-api-access-rh2hh\") pod \"perf-node-gather-daemonset-fblxd\" (UID: \"30cd1176-6c6e-4eb7-bc09-6d356e5e04a1\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:13.666376 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.666320 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30cd1176-6c6e-4eb7-bc09-6d356e5e04a1-sys\") pod \"perf-node-gather-daemonset-fblxd\" (UID: \"30cd1176-6c6e-4eb7-bc09-6d356e5e04a1\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:13.666376 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.666368 5084 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30cd1176-6c6e-4eb7-bc09-6d356e5e04a1-lib-modules\") pod \"perf-node-gather-daemonset-fblxd\" (UID: \"30cd1176-6c6e-4eb7-bc09-6d356e5e04a1\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:13.666523 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.666426 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30cd1176-6c6e-4eb7-bc09-6d356e5e04a1-sys\") pod \"perf-node-gather-daemonset-fblxd\" (UID: \"30cd1176-6c6e-4eb7-bc09-6d356e5e04a1\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:13.666523 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.666513 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30cd1176-6c6e-4eb7-bc09-6d356e5e04a1-lib-modules\") pod \"perf-node-gather-daemonset-fblxd\" (UID: \"30cd1176-6c6e-4eb7-bc09-6d356e5e04a1\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:13.724372 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.724284 5084 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh2hh\" (UniqueName: \"kubernetes.io/projected/30cd1176-6c6e-4eb7-bc09-6d356e5e04a1-kube-api-access-rh2hh\") pod \"perf-node-gather-daemonset-fblxd\" (UID: \"30cd1176-6c6e-4eb7-bc09-6d356e5e04a1\") " pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:13.843170 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:13.843132 5084 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:14.216699 ip-10-0-141-84 kubenswrapper[5084]: W1103 23:17:14.216670 5084 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod30cd1176_6c6e_4eb7_bc09_6d356e5e04a1.slice/crio-c88c40eaa70fb3ab66215782d0cca13647a220b69693bb0457a284378e91baf4 WatchSource:0}: Error finding container c88c40eaa70fb3ab66215782d0cca13647a220b69693bb0457a284378e91baf4: Status 404 returned error can't find the container with id c88c40eaa70fb3ab66215782d0cca13647a220b69693bb0457a284378e91baf4 Nov 03 23:17:14.219090 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:14.219052 5084 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd"] Nov 03 23:17:14.313227 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:14.313177 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-v5q4x_243c55d2-1d9c-4e22-be54-9360d9f506f3/manager/0.log" Nov 03 23:17:15.045269 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:15.045229 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" event={"ID":"30cd1176-6c6e-4eb7-bc09-6d356e5e04a1","Type":"ContainerStarted","Data":"5aa8af2ea35324f008a40f0e6e6411e690449379fc57375ffb0218a3cc58bebb"} Nov 03 23:17:15.045980 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:15.045290 5084 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" event={"ID":"30cd1176-6c6e-4eb7-bc09-6d356e5e04a1","Type":"ContainerStarted","Data":"c88c40eaa70fb3ab66215782d0cca13647a220b69693bb0457a284378e91baf4"} Nov 03 23:17:15.045980 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:15.045383 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:15.163217 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:15.163158 5084 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" podStartSLOduration=2.163142445 podStartE2EDuration="2.163142445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:17:15.161142031 +0000 UTC m=+4202.755174006" watchObservedRunningTime="2026-11-03 23:17:15.163142445 +0000 UTC m=+4202.757174592" Nov 03 23:17:20.882184 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:20.882149 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-8gtxf_ab2929c7-ab47-4660-80eb-f973bf6efdae/cluster-monitoring-operator/0.log" Nov 03 23:17:21.062256 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:21.062224 5084 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9px9d/perf-node-gather-daemonset-fblxd" Nov 03 23:17:26.029211 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:26.029174 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xcffl_36630490-7771-45f7-af2a-5d37a089f9e9/node-exporter/0.log" Nov 03 23:17:26.441047 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:26.440969 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xcffl_36630490-7771-45f7-af2a-5d37a089f9e9/kube-rbac-proxy/0.log" Nov 03 23:17:26.851212 ip-10-0-141-84 kubenswrapper[5084]: I1103 23:17:26.851175 5084 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xcffl_36630490-7771-45f7-af2a-5d37a089f9e9/init-textfile/0.log"